Zero-shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with varied visual manipulation intents across domains, scenes, objects, and attributes. A key challenge is that existing datasets contain limited intent-relevant annotations, making it hard for models to infer human intent from textual modifications. We introduce an intent-centric image–text dataset generated via reasoning by a Multimodal Large Language Model (MLLM) to better train ZS-CIR models for human manipulation intent understanding. Building on this dataset, we propose De-MINDS, a framework that distills the MLLM’s reasoning ability to capture manipulation intent and enhance models’ comprehension of modified text. A simple mapping network translates image information into language space and combines it with the manipulation text to form a query. De-MINDS then extracts intention-relevant information from this query and encodes it as pseudo-word tokens for accurate ZS-CIR. Across four ZS-CIR tasks, De-MINDS shows strong generalization and improves over existing methods by 2.15% to 4.05%, establishing new state-of-the-art results with comparable inference time.

The rapid development of Diffusion Language Models (DLMs) raises concerns about watermarking for DLM-generated detection.However, existing sequential LLM watermarking cannot be directly applied to DLMs, as DLMs' generation order is arbitrary.While emerging studies adapt biased LLM watermarking to DLMs by temporarily predicting the watermark prefix, they suffer from degraded quality and unstable watermarking due to bias accumulation and prediction errors.Besides, they cannot carry multi-bit watermarks.In this paper, we propose unbiased multi-bit watermarking for DLMs.We introduce a stability-aware constraint that allows watermarking only in stable contexts and a bit-controlled, unbiased modulation to preserve the original DLM output distribution, achieving stable watermarking with minimal quality impact.To enhance detection robustness, we design a Regret-based Remasking, which grants a "second chance" for unwatermarked tokens to be regenerated.It can seamlessly integrate into DLM inference with no added diffusion steps and latency.Experiments across DLMs and various tasks show that our scheme is effective, achieving superior generation quality compared to baselines while maintaining high detection accuracy and multi-bit capacity.Our code is available here.

Distributed Machine Learning (DML) enables parallel training but creates conflicts regarding model ownership determination, as verifying the specific contributions of individual clients during weight updates remains a challenge. To resolve ownership disputes and prevent malicious clients from forging data, verifying the integrity and effectiveness of each client's unique training history can fairly establish their partial ownership of the final DML model. In this paper, for the first time we propose a novel Matrix Commitment-based DML Ownership Verification (MAMMON) scheme that adopts a concise proof to ensure computational integrity and correctness within a limited computing cost. In contrast to computationally intensive SNARK-based approaches, MAMMON constructs a multi-linear tree structure to reduce proof update costs and avoids complex arithmetic circuit computations by standardizing the training process. Additionally, we watermark the weight proofs using client identity private keys to safeguard the commitments against tampering or unauthorized acquisition. Extensive experiments demonstrate MAMMON's superior performance in preserving computational integrity of DML training, covering both theoretical analysis and comparisons with other existing baselines.

Distributed Ledger Technology (DLT) has emerged as a promising solution for constructing decentralized and trustworthy platforms in recent years. Although blockchain, as a form of distributed ledger technology known for its tamper resistance and trustworthiness, seems to be an ideal solution for addressing trust issues, common blockchain systems face limitations due to the ever-growing volume of data. In this paper, we propose a dynamic storage scheme calledEfficiency-awareLow-storageScalability (ELS), which allocates variable data storage for the distributed ledgers. Our approach is designed to provide load balancing and security, allowing dynamic nodes to join or exit the network seamlessly. We utilize graph theory to group storage nodes in the blockchain by determining their network distances. These groups are then mapped into points in a two-dimensional virtual plane, meaning that block storage depends on the coordinates within the virtual plane. To balance storage performance with the evolving block allocation strategy, we develop a Voronoi diagram-based method to achieve near real-time adoptions that meet dynamic allocation requirements in blockchain systems. Experimental evaluations have demonstrated that our approach can reduce storage requirements by approximately 93% at each node, compared to full replication in 10000 block chains.

VQA models, which answer questions about images by combining both visual and textual information, have been proven susceptible to adversarial attacks. These attacks introduce subtle perturbations to the input data to manipulate the model’s predictions. This paper focuses on adversarial attacks targeting VQA models that follow the “pre-training & fine-tuning” paradigm, an area that remains under-explored. We have identified two key issues in the current field. On one hand, existing multi-modal attacks have low ASR due to inter-modal semantic inconsistency from insufficient cross-modal interaction. On the other hand, the dilemma between attack effectiveness and stealthiness limits the practical applicability of adversarial texts. To address these issues, we propose GIGAS, an innovative attack that uses multi-modal generative models to explore multi-modal interaction through three key modules tailored to solve above-mentioned problems. MIGA aligns adversarial visual features with semantics of misleading images generated by multi-modal generative models to mitigate cross-modal inconsistencies. GSA employs MLLMs to generate natural adversarial texts with greater variation and evaluate similarity to filter based on clean images, balancing effectiveness and stealthiness. Iteration Allocation dynamically adjusts attack iterations based on image-text similarity, maximizing the utility of the limited iterations. Experiments conducted on various VL models and VQA datasets demonstrate superior attack performance, with an average ASR of 89.09% on VQAv2.0. Furthermore, our GIGAS exhibits outstanding transferability, around 60% ASR, across diverse models and specific domains. Our code will be available at: https://github.com/Yvonna-cloud/GIGAS.

Recent booming development of Generative Artificial Intelligence (GenAI) has facilitated model commercialization to reinforce the model performance, including licensing or trading Deep Neural Network (DNN) models. However, DNN model trading may violate the benefit of the model owner due to unauthorized replications or misuse of the model. Model identity auditing is a challenging issue in protecting DNN model ownership, and verifying the integrity and ownership of models is one of the critical obstacles. In this paper, we focus on the above issue and propose anAccumulator-enabledAuditing forDecentralizedIdentity of DNNModel (A2-DIDM) that utilizes blockchain and zero-knowledge techniques to protect data and function privacy while ensuring the lightweight on-chain ownership verification. The proposed model presents a scheme of identity records via configuring model weight checkpoints with zero-knowledge proofs, which incorporates predicates to capture incremental state changes in model weight checkpoints. Our scheme ensures both computational integrity and programmability in DNN training process so that the uniqueness of the weight checkpoint sequence in a DNN model is preserved. A2-DIDM also addresses privacy protections in decentralized identity. We systematically analyze the security and robustness of our proposed model and further evaluate the effectiveness and usability of auditing DNN model identities. The code is available athttps://github.com/xtx123456/A2-DIDM.git.

Smart contracts play a pivotal role in blockchain ecosystems, and fuzzing remains a critical approach to securing them. However, existing smart contract fuzzers often optimize either seed generation or mutation scheduling in isolation and rely on narrow, fragmented feedback signals, leaving multi-transaction reasoning and stagnation recovery under-explored. In this work, we propose aLarge Language Models(LLMs)-based Multi-feedback Smart Contract Fuzzing framework (LLAMA). Key components of the proposed LLAMA include: (i) a hierarchical prompting strategy that guides LLMs to generate structurally valid, context-aware multi-transaction initial seeds, together with a lightweight pre-fuzzing phase that validates and prioritizes high-potential LLM-generated candidates; (ii) a multi-feedback-guided evolutionary optimization module that jointly optimizes seed selection and mutation scheduling by a group of constraints for driving an LLM-bootstrapped bandit scheduler. (iii) an LLM-guided hybrid fuzzing module that integrates evolutionary fuzzing with a dual-channel recovery mechanism, which concurrently employs asynchronous coverage-stagnation- based LLM reseeding and selective symbolic execution to resolve complex path constraints. Our extensive experiments demonstrate that LLAMA outperforms state-of-the-art fuzzers in both coverage and vulnerability detection. Specifically, it achieves 92% instruction coverage on small contracts and 81% on large contracts, while detecting 132 out of 148 known vulnerabilities across diverse categories. Ablation studies further evidence that the proposed multi-feedback and hybrid recovery strategies have strong impact on LLAMA’s performance. The results explain LLAMA’s effectiveness, adaptability, and practicality in complex smart contract scenarios.

Sharing relational databases is essential in today’s data-driven world for fostering collaboration, enhancing efficiency, and enabling real-time data access. However, privacy and copyright issues arise when sharing privacy-sensitive or valuable data. Additionally, high utility is required in shared data to enable accurate data mining and analysis. Entry-level differentially private fingerprinting schemes (DPFS) could address these concerns. In a DPFS, data can be securely shared without leaking original values while still supporting accurate analysis. Moreover, detectable fingerprints can deter unauthorized redistribution. However, existing DPFSs often lack utility—due to format changes and entry-wise bias—or robustness, as fingerprints can be removed undetected. In this paper, we propose an unbiased and robust differential privacy-based fingerprinting scheme (DPFS), which ensures that the fingerprinted copy remains an unbiased estimate of the original data. By incorporating differential privacy noise, our scheme effectively mitigates alteration, collusion, and hybrid attacks. Our DPFS satisfies ϵ-entry-level differential privacy, enabling clients to conduct unbiased analysis. To improve robustness, we design group-based fingerprint detection, which estimates the mean of injected noise per group with error tolerance. We provide a theoretical robustness analysis and propose a method for achieving optimal robustness. Experiments on four real-world databases show that our scheme consistently detects fingerprints and improves accuracy by up to 20% on machine learning tasks compared to existing DPFSs.

Multi-modal Federated Learning (MFL) is a distributed machine learning paradigm that enables multiple participants with multi-modal data to collaboratively train a global model for multi-modal tasks without sharing their local data. MFL typically deploys the trained global model as an Embedding-as-a-Service (EaaS), allowing participants to obtain embeddings for downstream tasks. However, it increases the risk of unauthorized copying and leakage of the model. Protecting the ownership of the MFL model while maintaining model performance is challenging. In this paper, we propose the first general model ownership protection framework for MFL, named MFL-Owner. MFL-Owner decouples the watermarking process from the model training process and addresses both ownership verification and traceability, effectively safeguarding the interests of the MFL collective. MFL-Owner leverages the concept of orthogonal transformations by incorporating a linear transformation matrix with orthogonal constraints into the model, achieving high-quality ownership verification and traceability with minimal impact on model performance. To enhance the practicality of the watermark and prevent conflicts among multiple clients during tracing, we propose a trigger dataset selection method based on out-of-distribution data combined with Gaussian noise perturbation. Our experiments on multiple datasets demonstrate that MFL-Owner is effective for model ownership verification and traceability for MFL.

Vision-language models (VLMs) have made significant progress in image classification by training with large-scale paired image-text data. Their performances largely depend on the prompt quality. While recent methods show that visual descriptions generated by large language models (LLMs) enhance the generalization of VLMs, class-specific prompts may be inaccurate or lack discrimination due to the hallucination in LLMs. In this paper, we aim to find visually discriminative prompts for fine-grained categories with minimal supervision and no human-in-the-loop. An evolution-based algorithm is proposed to progressively optimize language prompts from task-specific templates to class-specific descriptions. Unlike optimizing templates, the search space shows an explosion in class-specific candidate prompts. This increases prompt generation costs, iterative times, and the overfitting problem. To this end, we first introduce several simple yet effective edit-based and evolution-based operations to generate diverse candidate prompts by one-time query of LLMs. Then, two sampling strategies are proposed to find a better initial search point and reduce traversed categories, saving iteration costs. Moreover, we apply a novel fitness score with entropy constraints to mitigate overfitting. In a challenging one-shot image classification setting, our method outperforms existing textual prompt-based methods and improves LLM-generated description methods across 13 datasets. Meanwhile, we demonstrate that our optimal prompts improve adapter-based methods and transfer effectively across different backbones. Our code is available at here.

Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent across domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to modify a reference image according to manipulation text to accurately retrieve a target image, especially when the reference image is missing essential target content. In this paper, we propose a novel prediction-based mapping network, named PrediCIR, to adaptively predict the missing target visual content in reference images in the latent space before mapping for accurate ZS-CIR. Specifically, a world view generation module first constructs a source view by omitting certain visual content of a target view, coupled with an action that includes the manipulation intent derived from existing image-caption pairs. Then, a target content prediction module trains a world model as a predictor to adaptively predict the missing visual information guided by user intention in manipulating text at the latent space. The two modules map an image with the predicted relevant information to a pseudo-word token without extra supervision. Our model shows strong generalization ability on six ZS-CIR tasks. It obtains consistent and significant performance boosts ranging from 1.73% to 4.45% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://github.com/Pter61/predicir.

Composed Image Retrieval (CIR) aims to retrieve target images that closely resemble a reference image while integrating user-specified textual modifications, thereby capturing user intent more accurately. Existing training-free zero-shot CIR (ZS-CIR) methods often employ a two-stage process: they first generate a caption for the reference image and then use Large Language Models for reasoning a target description. However, these methods suffer from missing critical visual details and limited reasoning capabilities, leading to suboptimal retrieval performance. To address these challenges, we propose a novel, training-free one-stage method, One-Stage Reflective Chain-of-Thought Reasoning (OSrCIR) for ZS-CIR, which employs Multimodal Large Language Models to retain essential visual information in a single-stage reasoning process, eliminating the information loss in two-stage methods. Our Reflective Chain-of-Thought framework further improves interpretative accuracy by aligning manipulation intent with contextual cues from reference images. OSrCIR achieves performance gains of 1.80% to 6.44% over existing training-free methods across multiple tasks, setting new state-of-the-art results in ZS-CIR and enhancing its utility in vision-language applications. Our code is available at https://github.com/microsoft/ACV/tree/main/OSrCIR.

Latent diffusion models have exhibited considerable potential in generative tasks. Watermarking is considered to be an alternative to safeguard the copyright of generative models and prevent their misuse. However, in the context of model distribution scenarios, the accessibility of models to large scale of model users brings new challenges to the security, efficiency and robustness of existing watermark solutions. To address these issues, we propose a secure and efficient watermarking solution. A new security mechanism is designed to prevent watermark leakage and watermark escape, which considers watermark randomness and watermark-model association as two constraints for mandatory watermark injection. To reduce the time cost of training the security module, watermark injection and the security mechanism are decoupled, ensuring that fine-tuning VAE only accomplishes the security mechanism without the burden of learning watermark patterns. A watermark distribution-based verification strategy is proposed to enhance the robustness against diverse attacks in the model distribution scenarios. Experimental results prove that our watermarking consistently outperforms existing six baselines on effectiveness and robustness against ten image processing attacks and adversarial attacks, while enhancing security in the distribution scenarios. The code is available at https://anonymous.4open.science/r/DistriMark-F11F/.

Privacy-preserving data publishing has garnered widespread concern. Anonymization as a kind of privacy protection technique can safeguard individual privacy. In the context of publishing data, benign data updates occur frequently in the associated legitimate administrations and applications. In order to address updates, dynamic anonymization algorithms have been proposed. However, existing algorithms are unable to adequately generalize datasets, resulting in higher information loss. Furthermore, none of these algorithms involves attribute updates so far, and they only focus on record updates. This paper designs an anonymization algorithm, MSAnony, which is the first to address attribute updates problem. The algorithm introduces a novel data structure called Generalization Tag for efficiently re-generalizing. Moreover, it supports dynamic insertion, deletion and modification of records and attributes by basic operations Merging and Split. In a real-world dataset, MSAnony has good data utility compared with Mondrian, Incognito, Flash and Clustering under record update scenarios. Moreover, it can address attribute insertion and deletion under different parameters.

Online market platforms serve as a central hub for sharing and deploying AI models among researchers, developers, and companies. In this context, watermarking techniques are essential to protect intellectual property (IP), preventing unauthorized use and duplication of large language models (LLMs). Two key challenges arise: (i) These platforms host diverse LLMs, yet current watermarking techniques are only tailored to specific models, such as fine-tuned or quantized LLMs. (ii) Efficient watermarking is critical. However, traditional methods require substantial data and costly hardware, which limits their feasibility. In this paper, we propose an efficient white-box LLM watermarking technique called ELLMark. This method treats LLMs as multi-layered matrices while embedding watermarks only relies on modifying the model's weights. To preserve LLMs' performance, it filters weights by correlations with the activation magnitudes and downstream tasks, then modifies weights as minimal as possible via histogram modulation. Notably, all phases are training-free with low hardware resources, making it efficient for online platforms. We conduct extensive experiments to evaluate the effectiveness of ELLMark on LLaMA-3, OPT, and Phi-3 LLMs. The results demonstrate that it achieves 100% success in watermark detection while preserving model performance. Moreover, the preprocessing, encoding, and decoding processes remain efficient, taking less than 7 minutes, 12 minutes, and 18 seconds, respectively, for models with 80B parameters. Lastly, it exhibits robustness against parameter overwriting, re-watermarking, forging, fine-tuning, and pruning attacks.

Abstract In order to share data without revealing private information, privacy-preserving data publishing techniques are proposed. K-anonymity and l-diversity secure against identity and attribute disclosure. Anonymization algorithms enforce the above models and are willing to reach two primary goals: achieving the privacy objective while maximizing data utility. Even though anonymization has been studied for decades, finding efficient techniques to improve data utility is an open question. It is a crucial challenge that impacts many anonymized data on the web, cloud, and IoT environments. However, some factors incur huge information loss for existing works. The original dataset may be transformed into generalized data to an excessive extent. To address this problem, we give a new framework and propose a heuristic algorithm called UnitAnony. It builds a full-coverage hierarchy for more generalization candidates and proposes an interval-mapping technique for fine-grained generalization extent. However, these improvements raise another challenge. It is the vast cost because more generalization will derive many operations for forming new values, grouping records, and verifying anonymization models. Therefore, we designed a data structure unit to generalize records at low costs and implemented a skipping strategy to execute the algorithm within an acceptable time. Besides, our algorithm can support many models. By evaluating well-known K-anonymity and l-diversity on real-world datasets, i.e., Adult and Census datasets, the experimental results demonstrate that our algorithm outperforms the existing algorithms (e.g., Mondrian, Top-down, Improved-Clustering, Flash, and Incognito) regarding data utility and effectiveness.

Blockchain-based decentralized identity (DID) typically employs identity aggregation techniques to support efficient and trustworthy identity authentication in order to meet the requirements of the high volume of service requests in Internet of Things (IoT). Due to the lack of effective mechanisms for heterogeneous DID (H-DID) aggregation, a complete aggregated identity authentication often requires multiple rounds of signature verification for different identity attributes. However, this setting brings trust and privacy issues, and one notable threat is the potential disclosure of secret identity information through the linkage of heterogeneous identity attributes when enormous IoT devices/accesses are involved. In this article, we focus on trustworthy authentication of DID and propose a novel anonymous verifiable credential-based aggregation for h-DID (AVCA-hDID). Our AVCA-hDID model supports anonymous ownership verification of DIDs through label randomization, thereby effectively safeguarding identity privacy in IoT. AVCA-hDID involves identifier aggregation and attribute aggregation for H-DIDs, ensuring both authentication efficiency and balancing trustworthiness and adoptability. We analyze the security and unlinkablility of our proposed model and further experiment evaluation demonstrates the efficiency and effectiveness of AVCA-hDID.

Ensuring scalability in cryptocurrency systems is significant in guaranteeing real-world utility along with the remarkable increment of cryptographic currency. As an alternative in solving scalability issue, payment channel allows users to deliver extensive offline transactions without uploading massive transaction details to the blockchain, such that increasing efficiency can be achieved. However, the implementation of payment channel still encounters privacy concerns when considering the publicly available transaction amounts and the potentials in mining associations between transaction parties. In this paper, we propose a novel payment channel scheme, entitledCommitment-basedAnonymousPayment ChannEl (CAPE), to facilitate unlimited off-chain bidirectional payments while guaranteeing participants’ privacy. The proposed scheme adopts zero-knowledge proof (zk-SNARKs) and verifiable timed (VTD) commitments to ensure the anonymity of the relationship between on-chain and off-chain transactions, privacy of transaction amounts, and security of balances. We comprehensively formalize security definitions and present rigorous proofs for each security attribute. Experiment results further demonstrate the practical viability of CAPE.

Multi-Party Computation (MPC), as a distributed computing paradigm, is considered to be a potential solution for providing privacy-preserving for applications following the client-server model. However, traditional MPC solutions cannot satisfy the publicly verifiable requirement of the client-server model. In this paper, we propose a blockchain-based verifiable MPC solution using Pedersen's threshold secret sharing and Lifted ElGamal encryption. We first build a data distribution method using Pedersen's threshold secret sharing and symmetric encryption to protect the privacy of inputs while ensuring robustness. Then, we propose a result processing algorithm using Lifted ElGamal encryption to safeguard the privacy of the outputs. Finally, we employ non-interactive zero-knowledge proof and Pedersen commitment to publicly verify the correctness of the encrypted outputs in the smart contract, enabling the detection of malicious parties. Theoretical analysis indicates that the proposed method can publicly verify the correctness of outputs without revealing plain-text inputs and outputs, which satisfy the privacy-preserving requirements of the client-server model. Experimental evaluations have demonstrated that our proposed approach is efficient regarding computation overhead, communication overhead, and response time in the output verification phase while achieving stronger privacy and robustness.

As decentralized storage becomes more prevalent, ensuring data integrity within these systems has gained increasing importance. However, the introduction of a centralized and trustedCloud Combiner(CO) to manage auditing across distributedCloud Servers(CSs) presents several challenges, which not only increases communication costs with independent CSs but also diminishes the credibility of the auditing results. Additionally, traditional approaches often require either certificate and key management or the establishment of secure transmission channels, which incur substantial overhead. Furthermore, the computational cost for index generation to support searchable auditing is often high. To address these challenges, we propose a collaborative and searchable integrity auditing scheme for multi-replica data that eliminates the need for a CO. Specifically, we design a certificate-based authenticator for multi-copy data blocks, which simplifies certificate and key management while removing the need for secure channels. To enable efficient searchable auditing, we introduce a novel index tag that matches authenticators, thereby reducing the computational overhead. Moreover, we leverage blockchain technology to enhance the credibility of the auditing process and ensure the integrity of the search results. Both security and performance analyses demonstrate that the proposed scheme is secure and efficient.

Federated Learning (FL) realizes distributed machine learning training via sharing model updates rather than raw data, thus ensuring data privacy. However, an attacker may infer the client's local original data from the model parameter so that original data leakage can be caused. While Differential Privacy (DP) is designed to address data leakage issues in FL, injecting noises during training reduces model accuracy. To minimize the negative impact caused by noises on model accuracy while considering privacy protections, in this article we propose an adaptive FL model, entitledRDP-basedAdaptiveFederatedLearning inShuffle model (RAFLS). To ensure the dataset privacy of clients, we inject adaptive noises into the client's local model by leveraging the adaptive layer-wise adaptive sensitivity of the local model. Our approach shuffles all local model parameters in order to address privacy explosion concerns caused by high-dimensional aggregation and multiple iterations. We further propose a fine-grained model weight aggregation scheme to aggregate all local models and obtain a global model. Our experiment evaluations demonstrate the proposed RAFLS has a better performance than the state-of-the-art methods in reducing noise's impact on model accuracy while protecting data, i.e., showing that the accuracy of RAFLS increases by 1.54% than that of the baseline scheme when$\epsilon = 2.0$and FashionMNIST under IID setting.

Vertical Federated Learning(VFL) has gained widespread attention due to its ability of enabling collaborative model training among participants with diverse data features.Differential Privacy(DP) offers provable privacy guarantees for VFL, but existing DP-based methods typically compromise accuracy for privacy protection. To address this issue, we propose a novel scheme, called Adaptive Differential Privacy-based Vertical Federated Learning (Ada-VFed), that enhances privacy of data features and labels by adding Gaussian noise separately to the transmitted intermediate results and gradients. To improve model accuracy, we incorporate adaptive constraints through regularization terms in the objective function to mitigate the impact of clipping operations. In addition, we propose a dynamic noise injection mechanism that adjusts noise according to the importance of each dimension, thereby balancing privacy protection and model accuracy. Our theoretical analysis provides privacy guarantees and convergence insights. Extensive experiments demonstrated that our scheme significantly outperforms state-of-the-art DP-based VFL methods in terms of accuracy. Even with a small privacy budget (e.g.,ϵ = 0.5), our method improves the accuracy on MNIST, FashionMNIST, and CIFAR-10 by 13.01%, 10.08%, and 3.40%, respectively, compared to traditional DP-based VFL methods.

Federated Learning (FL) is an alternative approach that facilitates training machine learning models on distributed users’ data while preserving privacy. However, clients have different local model structures and most local data are non-independent and identically distributed, so that FL encounters heterogeneity and catastrophic forgetting issues when clients continuously accumulate new knowledge. In this work, we propose a scheme called MUFTI (Multi-Domain Distillation-based Heterogeneous Federated ConTInuous Learning). On one hand, we have extended domain adaptation to FL via extracting features to obtain feature representations on unlabeled public datasets for collaborative training, narrowing the distance between feature outputs of different models under the same sample. On the other hand, we propose a combining knowledge distillation method to solve the catastrophic forgetting issue. Within a single task, dual-domain distillation is used to avoid data forgetting between different domains; for cross task learning in task flow, the logits output of the previous model is used as the teacher to avoid forgetting old tasks. The experiment results showed that MUFTI had a better performance in accuracy and robustness comparing to state-of-the-art methods. The evaluation also demonstrated that MUFTI could perform well in handling task increment issues, reducing catastrophic forgetting, and achieving trade-offs between multiple objectives.

As a collaborative framework designed to safeguard privacy,Federated Learning(FL) seeks to protect participants’ data throughout the training process. However, the framework still faces security risks from poisoning attacks, arising from the unmonitored process of client-side model updates. Most existing solutions address scenarios where less than half of clients are malicious, i.e., which leaves a significant challenge to defend against attacks when more than half of participants are malicious. In this paper, we propose a FL scheme, named FedAMM, that resists backdoor attacks across various data distributions and malicious client ratios. We develop a novel backdoor defense mechanism to filter out malicious models, aiming to reduce the performance degradation of the model. The proposed scheme addresses the challenge of distance measurement in high-dimensional spaces by applyingPrincipal Component Analysis(PCA) to improve clustering effectiveness. We borrow the idea of critical parameter analysis to enhance discriminative ability in non-iid data scenarios, via assessing the benign or malicious nature of models by comparing the similarity of critical parameters across different models. Finally, our scheme employs a hierarchical noise perturbation to improve the backdoor mitigation rate, effectively eliminating the backdoor and reducing the adverse effects of noise on task accuracy. Through evaluations conducted on multiple datasets, we demonstrate that the proposed scheme achieves superior backdoor defense across diverse client data distributions and different ratios of malicious participants. With 80% malicious clients, FedAMM achieves low backdoor attack success rates of 1.14%, 0.28%, and 5.53% on MNIST, FMNIST, and CIFAR-10, respectively, demonstrating enhanced robustness of FL against backdoor attacks.

Encrypted traffic classification is highly challenging in network security due to the need for extracting robust features from content-agnostic traffic data. Existing approaches face critical issues: (i) Distribution drift, caused by reliance on the closedworld assumption, limits adaptability to real-world, shifting patterns; (ii) Dependence on labeled data restricts applicability where such data is scarce or unavailable. Large language models (LLMs) have demonstrated remarkable potential in offering generalizable solutions across a wide range of tasks, achieving notable success in various specialized fields. However, their effectiveness in traffic analysis remains constrained by challenges in adapting to the unique requirements of the traffic domain. In this paper, we introduce a novel traffic representation model named Encrypted Traffic Out-of-Distribution Instruction Tuning with LLM (ETooL), which integrates LLMs with knowledge of traffic structures through a self-supervised instruction tuning paradigm. This framework establishes connections between textual information and traffic interactions. ETooL demonstrates more robust classification performance and superior generalization in both supervised and zero-shot traffic classification tasks. Notably, it achieves significant improvements in F1 scores: APP53 (I.I.D.) to 93.19%(6.62%↑) and 92.11%(4.19%↑), APP53 (O.O.D.) to 74.88%(18.17%↑) and 72.13%(15.15%↑), and ISCX-Botnet (O.O.D.) to 95.03%(9.16%↑) and 81.95%(12.08%↑). Additionally, we construct NETD, a traffic dataset designed to support dynamic distributional shifts, and use it to validate ETooL’s effectiveness under varying distributional conditions. Furthermore, we evaluate the efficiency gains achieved through ETooL’s instruction tuning approach.

Provable Data Possession (PDP) is an alternative technique that guarantees the integrity of remote data. However, most current PDP schemes are inapplicable to similarity-like data checking with the same attribute, i.e., when there are numerous similar files to be checked by Data Owners (DOs). Some traditional models cannot resist the corrupt auditors who always generate biased challenge information. Besides, a copy-summation attack exists in some schemes, which means the Cloud Server (CS) can bypass the verification by storing the median value instead of initial data via summation operation. To address the issues above, in this work, we propose a keyword searchable PDP scheme for large similar data checking. To achieve searchability, we introduce the notion of a keyword in PDP and design a specific index structure to match the authenticator. The scheme enables all matched files to be auditable and verifiable, while guaranteeing privacy protections. Unlike existing methods, our Third Party Auditor (TPA) checks all similar data containing the same keyword simultaneously. We utilize unpredictable yet verifiable public information on the blockchain to generate challenge information, rather than relying on a centralized TPA. The proposed scheme can resist copy-summation attacks. Theoretical analysis demonstrates that the proposed scheme satisfies the security requirements, and our evaluations demonstrate its efficiency.

Vertical federated learning (VFL) provides a privacy-preserving method for machine learning, enabling collaborative training across multiple institutions with vertically distributed data. Existing VFL methods assume that participants passively gain local models of the same structure and communicate with active pary during each training batch. However, due to the heterogeneity of participating institutions, VFL with heterogeneous models for efficient communication is indispensable in real-life scenarios. To address this challenge, we propose a new VFL method called Practical Heterogeneous Vertical Federated Learning via Representation Learning (PraVFed) to support the training of parties with heterogeneous local models and reduce communication costs. Specifically, PraVFed employs weighted aggregation of local embedding values from the passive party to mitigate the influence of heterogeneous local model information on the global model. Furthermore, to safeguard the passive party’s local sample features, we utilize blinding factors to protect its local embedding values. To reduce communication costs, the passive party performs multiple rounds of local pre-model training while preserving label privacy. We conducted a comprehensive theoretical analysis and extensive experimentation to demonstrate that PraVFed reduces communication overhead under heterogeneous models and outperforms other approaches. For example, when the target accuracy is set at 60% under the CINIC10 dataset, the communication cost of PraVFed is reduced by 70.57% compared to the baseline method. Our code is available athttps://github.com/wangshuo105/PraVFed_main.

Vertical Federated Learning (VFL) allows collaborative machine learning without sharing local data. However, existing VFL methods face challenges when dealing with heterogeneous local models among participants, which affects optimization convergence and generalization of participants' local knowledge aggregation. To address this challenge, this paper proposes a novel approach calledEmbeddingAggregation-based HeterogeneousModelsTraining in Vertical Federated Learning(EASTER). EASTER focuses on aggregating the local embeddings of each participant's knowledge during forward propagation. We propose an embedding protection method based on lightweight blinding factors, which injects the blinding factors into the local embedding of the passive party. However, the passive party does not own the sample labels, so the local model's gradient cannot be calculated locally. To overcome this limitation, we propose a new method in which the active party assists the passive party in computing its local heterogeneous model gradients. Theoretical analysis and extensive experiments demonstrate that EASTER can simultaneously train multiple heterogeneous models and outperform some recent methods in model performance. For example, compared with the state-of-the-art method, the model accuracy of EASTER was improved by 7.22% under the CIFAR-10 dataset.

Different from the Composed Image Retrieval task that requires expensive labels for training task-specific models, Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent that could be related to domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to learn a more accurate image representation that has adaptive attention to the reference image for various manipulation descriptions. In this paper, we propose a novel context-dependent mapping network, named Context-I2W, for adaptively converting description-relevant Image information into a pseudo-word token composed of the description for accurate ZS-CIR. Specifically, an Intent View Selector first dynamically learns a rotation rule to map the identical image to a task-specific manipulation view. Then a Visual Target Extractor further captures local information covering the main targets in ZS-CIR tasks under the guidance of multiple learnable queries. The two complementary modules work together to map an image to a context-dependent pseudo-word token without extra supervision. Our model shows strong generalization ability on four ZS-CIR tasks, including domain conversion, object composition, object manipulation, and attribute manipulation. It obtains consistent and significant performance boosts ranging from 1.88% to 3.60% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://anonymous.4open.science/r/Context-I2W-4224/.

Along with the development of the biological industry, the system of biological assets governance has a higher-level requirement in asset identity verification due to the demands of digitization and Financial Technology (FinTech). In order to achieve broad scalability and adaptability in the governance of biological assets' identities, this work proposes a Biological Assets Decentralized Identity Management (BA-DID) system that develops a Decentralized Identity (DID)-based solution to addressing the verification issues in biological assets while considering multi-dimensional requirements. Blockchain technology is the fundamental infrastructure of the system. Authenticated certificates of the asset owners verify identity attributes. Financial Service Institutions (FSI) and governance agencies act as issuers, providing Verifiable Credentials (VC) for biological assets by using a group of attributes over a consensus. The asset owners hold VCs and can be available to other organizations as proof of the corresponding asset attributes. Our evaluations have demonstrated that the proposed approach has superior performance in biological assets verification, security, and maintenance.

Multi-task learning has shown large benefits in Natural Language Understanding (NLU). However, current state-of-the-arts (SOTAs) like MT-DNN and MMoE do not model task relationships explicitly and fail to obtain effective task alignment. In this paper, we propose a Collaborative Multi-Task Representation (CMTR) framework to tackle this problem. We capture instance-level task relations through a task interaction layer, which helps guide the fusion of task-oriented representations into the final representation. Moreover, tailored loss functions are proposed to facilitate the learning of task alignment. Specifically, we leverage knowledge distillation as an auxiliary loss to assist the adaptation layers in generating task-oriented representations. We also introduce a regularization loss to learn better gating functions for multi-task fusion. Empirically, CMTR outperforms SOTA multi-task learning frameworks on most natural language understanding tasks in the GLUE benchmark. Furthermore, it achieves better task alignment and demonstrates good interpretability.

Current text-video retrieval methods mainly rely on cross-modal matching between queries and videos to calculate their similarity scores, which are then sorted to obtain retrieval results. This method considers the matching between each candidate video and the query, but it incurs a significant time cost and will increase notably with the increase of candidates. Generative models are common in natural language processing and computer vision, and have been successfully applied in document retrieval, but their application in multimodal retrieval remains unexplored. To enhance retrieval efficiency, in this paper, we introduce a model-based video indexer named T2VIndexer, which is a sequence-to-sequence generative model directly generating video identifiers and retrieving candidate videos with constant time complexity. T2VIndexer aims to reduce retrieval time while maintaining high accuracy. To achieve this goal, we propose video identifier encoding and query-identifier augmentation approaches to represent videos as short sequences while preserving their semantic information. Our method consistently enhances the retrieval efficiency of current state-of-the-art models on four standard datasets. It enables baselines with only 30%-50% of the original retrieval time to achieve better retrieval performance on MSR-VTT (+1.0%), MSVD (+1.8%), ActivityNet (+1.5%), and DiDeMo (+0.2%). The code is available at https://github.com/Lilidamowang/T2VIndexer-generativeSearch.

Recent work shows that documents from encyclopedias serve as helpful auxiliary information for zero-shot learning. Existing methods align the entire semantics of a document with corresponding images to transfer knowledge. However, they disregard that semantic information is not equivalent between them, resulting in a suboptimal alignment. In this work, we propose a novel network to extract multi-view semantic concepts from documents and images and align the matching rather than entire concepts. Specifically, we propose a semantic decomposition module to generate multi-view semantic embeddings from visual and textual sides, providing the basic concepts for partial alignment. To alleviate the issue of information redundancy among embeddings, we propose the local-to-semantic variance loss to capture distinct local details and multiple semantic diversity loss to enforce orthogonality among embeddings. Subsequently, two losses are introduced to partially align visual-semantic embedding pairs according to their semantic relevance at the view and word-to-patch levels. Consequently, we consistently outperform state-of-the-art methods under two document sources in three standard benchmarks for document-based zero-shot learning. Qualitatively, we show that our model learns the interpretable partial association. Code is available at https://github.com/MorningStarOvO/EmDepart.

Blockchain is a revolutionary technology for constructing trustworthy communications for 6G multiaccess edge computing (6G-MEC). Designing a blockchain-based access control system for 6G-MEC is challenging due to the highly heterogeneous edge devices (EDs) in 6G-MEC. However, traditional blockchain-based access control methods cannot satisfy the heterogeneous device scenarios and are unable to assign voting weights based on the performance of EDs. In this article, we propose a trustworthy access control method for 6G-MEC networks and demonstrate how blockchain can be utilized in our proposed blockchain-assisted multiaccess control approach. Our approach develops an attribute validation and a validation weight method to offer strengthened access control in a decentralized context. We have proposed a trustworthy access control for multiaccess edge computing by using a multitier blockchain architecture and a reinforcement-learning-based weight determination algorithm for ED to eliminate the influence on blockchain consensus behavior. Experimental results demonstrate the effectiveness of our proposed approach.

Generating radiology reports automatically reduces the workload of radiologists and helps the diagnoses of specific diseases. Many existing methods take this task as modality transfer process. However, since the key information related to disease accounts for a small proportion in both image and report, it is hard for the model to learn the latent relation between the radiology image and its report, thus failing to generate fluent and accurate radiology reports. To tackle this problem, we propose a memory-based cross-modal semantic alignment model (MCSAM) following an encoder-decoder paradigm. MCSAM includes a well initialized long-term clinical memory bank to learn disease-related representations as well as prior knowledge for different modalities to retrieve and use the retrieved memory to perform feature consolidation. To ensure the semantic consistency of the retrieved cross modal prior knowledge, a cross-modal semantic alignment module (SAM) is proposed. SAM is also able to generate semantic visual feature embeddings which can be added to the decoder and benefits report generation. More importantly, to memorize the state and additional information while generating reports with the decoder, we use learnable memory tokens which can be seen as prompts. Extensive experiments demonstrate the promising performance of our proposed method which generates state-of-the-art performance on the MIMIC-CXR dataset.

Image-text retrieval requires the system to bridge the heterogenous gap between vision and language for accurate retrieval while keeping the network lightweight-enough for efficient retrieval. Existing trade-off solutions mainly study from the view of incorporating cross-modal interactions with the independent-embedding framework or leveraging stronger pre-trained encoders, which still demand time-consuming similarity measurement or heavyweight model structure in the retrieval stage. In this work, we propose an image-text alignment module SelfAlign on top of the independent-embedding framework, which improves the retrieval accuracy while maintains the retrieval efficiency without extra supervision. SelfAlign contains two collaborative sub-modules that force image-text alignment at both the concept level and context level by self-supervised contrastive learning. It doesn't require cross-modal embedding interactions during training while maintaining independent image and text encoders during retrieval. With comparable time cost, SelfAlign consistently boosts the accuracy of state-of-the-art non-pre-training independent-embedding models respectively by 9.1%, 4.2%, and 6.6% in terms of R@sum score on Flickr30 K, MS-COCO 1 K and MS-COCO 5 K datasets. The retrieval accuracy also outperforms most existing interactive-embedding models with orders of magnitude decrease in retrieval time. The source code is available at:https://github.com/Zjamie813/SelfAlign.

Vertical Federated Learning (VFL) effectively addresses the issue of data isolation, which makes data mining secure. Most VFL implementations rely on a single server or third party for training, which will be terminated if the server or third party fails. In addition, the model accuracy trained by VFL depends on the quality of the client’s local features; nevertheless, the client’s local feature quality is difficult to verify. There exists a chance that the features owned by the client are irrelevant to the model or the intermediate results submitted by the client are inaccurate, such that the model’s accuracy will be seriously affected. In order to solve the single point failure and model accuracy issues in VFL, this paper first proposes a Blockchain – based Decentralized VFL (BDVFL) training model. With the integration of blockchain and the VFL training process, the nodes within the blockchain are categorized into non-training and training nodes. Our method focuses on the scenario in which all training nodes possess labeled data and actively engage in the training procedure of VFL. To be specific, first, each client utilizes local features and initial models to carry out forward activation and generate intermediate results. Second, we randomly choose a training node and combine it with the intermediate results from all clients to formulate the loss function. Finally, each client updates the local model by using the gradient. To protect the raw features, a blinding factor is utilized for safeguarding the intermediate results submitted by the client, such that the training nodes cannot infer the local features from intermediate results. To mitigate the interference of irrelevant training outcomes from clients on the model’s accuracy, we propose a verifiable aggregation method to assess the validity of the intermediate results submitted by the clients. We have conducted both theoretical and experimental analysis, and the results demonstrate the effectiveness of the proposed method.

Attention-based neural networks, such as Transformers, have become ubiquitous in numerous applications, including computer vision, natural language processing, and time-series analysis. In all kinds of attention networks, the attention maps are crucial as they encode semantic dependencies between input tokens. However, most existing attention networks perform modeling or reasoning based on representations, wherein the attention maps of different layers are learned separately without explicit interactions. In this paper, we propose a novel and generic evolving attention mechanism, which directly models the evolution of inter-token relationships through a chain of residual convolutional modules. The major motivations are twofold. On the one hand, the attention maps in different layers share transferable knowledge, thus adding a residual connection can facilitate the information flow of inter-token relationships across layers. On the other hand, there is naturally an evolutionary trend among attention maps at different abstraction levels, so it is beneficial to exploit a dedicated convolution-based module to capture this process. Equipped with the proposed mechanism, the convolution-enhanced evolving attention networks achieve superior performance in various applications, including time-series representation, natural language understanding, machine translation, and image classification. Especially on time-series representation tasks, Evolving Attention-enhanced Dilated Convolutional (EA-DC-) Transformer outperforms state-of-the-art models significantly, achieving an average of 17% improvement compared to the best SOTA. To the best of our knowledge, this is the first work that explicitly models the layer-wise evolution of attention maps. Our implementation is available at https://github.com/pkuyym/EvolvingAttention.

Knowledge-based visual question answering requires the ability of associating external knowledge for open-ended cross-modal scene understanding. One limitation of existing solutions is that they capture relevant knowledge from text-only knowledge bases, which merely contain facts expressed by first-order predicates or language descriptions while lacking complex but indispensable multimodal knowledge for visual understanding. How to construct vision-relevant and explainable multimodal knowledge for the VQA scenario has been less studied. In this paper, we propose MuKEA to represent multimodal knowledge by an explicit triplet to correlate visual objects and fact answers with implicit relations. To bridge the heterogeneous gap, we propose three objective losses to learn the triplet representations from complementary views: embedding structure, topological relation and semantic space. By adopting a pretraining and fine-tuning learning strategy, both basic and domain-specific multimodal knowledge are progressively accumulated for answer prediction. We outperform the state-of-the-art by 3.35% and 6.08% respectively on two challenging knowledge-required datasets: OK-VQA and KRVQA. Experimental results prove the complementary benefits of the multimodal knowledge with existing knowledge bases and the advantages of our end-to-end framework over the existing pipeline methods. The code is available at https://github.com/AndersonStra/MuKEA.

Relational triplet extraction (RTE) is a fundamental task for automatically extracting information from unstructured text, which has attracted growing interest in recent years. However, it remains challenging due to the difficulty in extracting the overlapping relational triplets. Existing approaches for overlapping RTE, either suffer from exposure bias or designing complex tagging scheme. In light of these limitations, we take an innovative perspective on RTE by modeling it as a word linking problem that learns to link from subject words to object words for each relation type. To this end, we propose a simple but effective multi-task learning model, WLinker, which can extract overlapping relational triplets in an end-to-end fashion. Specifically, we perform word link prediction based on multi-level biaffine attention for leaning the word-level correlations under each relation type. Additionally, our model joint entity detection and word link prediction tasks by a multi-task framework, which combines the local sequential and global dependency structures of words in sentence and captures the implicit interactions between the two tasks. Extensive experiments are conducted on two benchmark datasets NYT and WebNLG. The results demonstrate the effectiveness of WLinker, in comparison with a range of previous state-of-the-art baselines.

Proving ownership rights on relational databases is an important issue. The robust watermarking technique could claim ownership by insertion information about the data owner. Hence, it is vital to improving the robustness of watermarking technique in that intruders could launch types of attacks to corrupt the inserted watermark. Furthermore, attributes are explicit and operable objectives to destroy the watermark. To my knowledge, there does not exist a comprehensive solution to resist attribute attack. In this paper, we propose a robust watermarking technique that is robust against subset and attribute attacks. The novelties lie in several points: applying the classifier to reorder watermarked attributes, designing a secret sharing mechanism to duplicate watermark independently on each attribute, and proposing twice majority voting to correct errors caused by attacks for improving the accuracy of watermark detection. In addition, our technique has features of blind, key-based, incrementally updatable, and low false hit rate. Experiments show that our algorithm is robust against subset and attribute attacks compared with AHK, DEW, and KSF algorithms. Moreover, it is efficient with running time in both insertion and detection phases.

Encrypted traffic classification requires discriminative and robust traffic representation captured from content-invisible and imbalanced traffic data for accurate classification, which is challenging but indispensable to achieve network security and network management. The major limitation of existing solutions is that they highly rely on the deep features, which are overly dependent on data size and hard to generalize on unseen data. How to leverage the open-domain unlabeled traffic data to learn representation with strong generalization ability remains a key challenge. In this paper, we propose a new traffic representation model called Encrypted Traffic Bidirectional Encoder Representations from Transformer (ET-BERT), which pre-trains deep contextualized datagram-level representation from large-scale unlabeled data. The pre-trained model can be fine-tuned on a small number of task-specific labeled data and achieves state-of-the-art performance across five encrypted traffic classification tasks, remarkably pushing the F1 of ISCX-VPN-Service to 98.9% (5.2%↑), Cross-Platform (Android) to 92.5% (5.4%↑), CSTNET-TLS 1.3 to 97.4% (10.0%↑). Notably, we provide explanation of the empirically powerful pre-training model by analyzing the randomness of ciphers. It gives us insights in understanding the boundary of classification ability over encrypted traffic. The code is available at: https://github.com/linwhitehat/ET-BERT.

It is important for task-oriented dialogue systems to discover the dialogue structure (i.e. the general dialogue flow) from dialogue corpora automatically. Previous work models dialogue structure by extracting latent states for each utterance first and then calculating the transition probabilities among states. These two-stage methods ignore the contextual information when calculating the probabilities, which makes the transitions between the states ambiguous. This paper proposes a conversational graph (CG) to represent deterministic dialogue structure where nodes and edges represent the utterance and context information respectively. An unsupervised Edge-Enhanced Graph Auto-Encoder (EGAE) architecture is designed to model local-contextual and global-structural information for conversational graph learning. Furthermore, a self-supervised objective is introduced with the response selection task to guide the unsupervised learning of the dialogue structure. Experimental results on several public datasets demonstrate that the novel model outperforms several alternatives in aggregating utterances with similar semantics. The effectiveness of the learned dialogue structured is also verified by more than 5\% joint accuracy improvement in the downstream task of low resource dialogue state tracking.

Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, Yunhai Tong. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

Question answering systems usually use keyword searches to retrieve potential passages related to a question, and then extract the answer from passages with the machine reading comprehension methods. However, many questions tend to be unanswerable in the real world. In this case, it is significant and challenging how the model determines when no answer is supported by the passage and abstains from answering. Most of the existing systems design a simple classifier to determine answerability implicitly without explicitly modeling mutual interaction and relation between the question and passage, leading to the poor performance for determining the unanswerable questions. To tackle this problem, we propose a Multi-Step Co-Interactive Relation Network (MCR-Net) to explicitly model the mutual interaction and locate key clues from coarse to fine by introducing a co-interactive relation module. The co-interactive relation module contains a stack of interaction and fusion blocks to continuously integrate and fuse history-guided and current-query-guided clues in an explicit way. Experiments on the SQuAD 2.0 and DuReader datasets show that our model achieves a remarkable improvement, outperforming the BERT-style baselines in literature. Visualization analysis also verifies the importance of the mutual interaction between the question and passage.

It is prevalent to utilize external knowledge to help machine answer questions that need background commonsense, which faces a problem that unlimited knowledge will transmit noisy and misleading information. Towards the issue of introducing related knowledge, we propose a semantic-driven knowledge-aware QA framework, which controls the knowledge injection in a coarse-to-careful fashion. We devise a tailoring strategy to filter extracted knowledge under monitoring of the coarse semantic of question on the knowledge extraction stage. And we develop a semantic-aware knowledge fetching module that engages structural knowledge information and fuses proper knowledge according to the careful semantic of questions in a hierarchical way. Experiments demonstrate that the proposed approach promotes the performance on the CommonsenseQA dataset comparing with strong baselines.

Transformer is a ubiquitous model for natural language processing and has attracted wide attentions in computer vision. The attention maps are indispensable for a transformer model to encode the dependencies among input tokens. However, they are learned independently in each layer and sometimes fail to capture precise patterns. In this paper, we propose a novel and generic mechanism based on evolving attention to improve the performance of transformers. On one hand, the attention maps in different layers share common knowledge, thus the ones in preceding layers can instruct the attention in succeeding layers through residual connections. On the other hand, low-level and high-level attentions vary in the level of abstraction, so we adopt convolutional layers to model the evolutionary process of attention maps. The proposed evolving attention mechanism achieves significant performance improvement over various state-of-the-art models for multiple tasks, including image classification, natural language understanding and machine translation.

Scene graphs are semantic abstraction of images that encourage visual understanding and reasoning. However, the performance of Scene Graph Generation (SGG) is unsatisfactory when faced with biased data in real-world scenarios. Conventional debiasing research mainly studies from the view of balancing data distribution or learning unbiased models and representations, ignoring the correlations among the biased classes. In this work, we analyze this problem from a novel cognition perspective: automatically building a hierarchical cognitive structure from the biased predictions and navigating that hierarchy to locate the relationships, making the tail relationships receive more attention in a coarse-to-fine mode. To this end, we propose a novel debiasing Cognition Tree (CogTree) loss for unbiased SGG. We first build a cognitive structure CogTree to organize the relationships based on the prediction of a biased SGG model. The CogTree distinguishes remarkably different relationships at first and then focuses on a small portion of easily confused ones. Then, we propose a debiasing loss specially for this cognitive structure, which supports coarse-to-fine distinction for the correct relationships. The loss is model-agnostic and consistently boosting the performance of several state-of-the-art models. The code is available at: https://github.com/CYVincent/Scene-Graph-Transformer-CogTree.

Website fingerprinting (WF) attacks pose a threat to privacy of web activity, especially on anonymity networks such as Tor. Recent studies show that the deep neural network (DNN) significantly improves the impact of website fingerprinting attacks. Especially, DNN-based attack undermines the existing defense methods which are mainly rely on the manually designed rule. In this paper, we present a novel defense that generates universal perturbation that can transform original examples to adversarial examples which is effectively defending against a specific WF model. The proposed defense is evaluated on state-of-the-art DNN attack over a public Tor traffic dataset. The experimental results show our adversarial example generation method performs better than the baseline methods. The proposed defense defeats all existing WF attacks based on deep neural networks with a low overhead. Comparing with state-of-the-art defenses such as Walkie-Talkie and WTF-PAD with a lower bound of 31% and 64% overheads, the proposed defense achieves identical defense performance with at least 50% bandwidth overhead saving.

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue task involves multiple rounds of dialogues which cover a broad range of visual content that could be related to any objects, relationships or high-level semantics. Thus one of the key challenges in Visual Dialogue task is to learn a more comprehensive and semantic-rich image representation that can adaptively attend to the visual content referred by variant questions. In this paper, we first propose a novel scheme to depict an image from both visual and semantic views. Specifically, the visual view aims to capture the appearance-level information in an image, including objects and their visual relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Furthermore, on top of such dual-view image representations, we propose a Dual Encoding Visual Dialogue (DualVD) module, which is able to adaptively select question-relevant information from the visual and semantic views in a hierarchical mode. To demonstrate the effectiveness of DualVD, we propose two novel visual dialogue models by applying it to the Late Fusion framework and Memory Network framework. The proposed models achieve state-of-the-art results on three benchmark datasets. A critical advantage of the DualVD module lies in its interpretability. We can analyze which modality (visual or semantic) has more contribution in answering the current question by explicitly visualizing the gate values. It gives us insights in understanding of information selection mode in the Visual Dialogue task. The code is available at https://github.com/JXZe/Learning_DualVD.

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue involves multiple questions which cover a broad range of visual content that could be related to any objects, relationships or semantics. The key challenge in Visual Dialogue task is thus to learn a more comprehensive and semantic-rich image representation which may have adaptive attentions on the image for variant questions. In this research, we propose a novel model to depict an image from both visual and semantic perspectives. Specifically, the visual view helps capture the appearance-level information, including objects and their relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Futhermore, on top of such multi-view image features, we propose a feature selection framework which is able to adaptively capture question-relevant information hierarchically in fine-grained level. The proposed method achieved state-of-the-art results on benchmark Visual Dialogue datasets. More importantly, we can tell which modality (visual or semantic) has more contribution in answering the current question by visualizing the gate values. It gives us insights in understanding of human cognition in Visual Dialogue.

Keeping the conversation consistent and avoiding its repetition are two key factors to construct an intelligent multi-turn knowledge-grounded dialogue system. Although some works tend to combine history with external knowledge such as personal background information to boost dialogue quality, they are prone to ignore the fact that incorporating the same knowledge multiple times into the conversation leads to repetition. The main reason is the lack of effective control over the use of knowledge on the conversation level. So we design a history-adaption knowledge incorporation mechanism to build an effective multi-turn dialogue model. Our proposed model addresses repetition by recurrently updating the knowledge from the conversation level and progressively incorporating it into the history step-by-step. And the knowledge-grounded history representation also enhances the conversation consistency. Experimental results show that our proposed model significantly outperforms several retrieval-based models on some benchmark datasets. The human evaluation demonstrates that our model can maintain conversation consistent and reduce conversation repetition.

We propose a novel Bi-directional Cognitive Knowledge Framework (BCKF) for reading comprehension from the perspective of complementary learning systems theory. It aims to simulate two ways of thinking in the brain to answer questions, including reverse thinking and inertial thinking. To validate the effectiveness of our framework, we design a corresponding Bi-directional Cognitive Thinking Network (BCTN) to encode the passage and generate a question (answer) given an answer (question) and decouple the bi-directional knowledge. The model has the ability to reverse reasoning questions which can assist inertial thinking to generate more accurate answers. Competitive improvement is observed in DuReader dataset, confirming our hypothesis that bi-directional knowledge helps the QA task. The novel framework shows an interesting perspective on machine reading comprehension and cognitive science.

Visual Question Answering (VQA) is a representative task of cross-modal reasoning where an image and a free-form question in natural language are presented and the correct answer needs to be determined using both visual and textual information. One of the key issues of VQA is to reason with semantic clues in the visual content under the guidance of the question. In this paper, we propose Scene Graph Convolutional Network (SceneGCN) to jointly reason the object properties and their semantic relations for the correct answer. The visual relationship is projected into a deep learned semantic space constrained by visual context and language priors. Based on comprehensive experiments on two challenging datasets: GQA and VQA 2.0, we demonstrate the effectiveness and interpretability of the new model.

Visual Dialogue task requires an agent to be engaged in a conversation with human about an image. The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation. In this paper, we propose a novel generative decoding architecture to generate high-quality responses, which moves away from decoding the whole encoded semantics towards the design that advocates both transparency and flexibility. In this architecture, word generation is decomposed into a series of attention-based information selection steps, performed by the novel recurrent Deliberation, Abandon and Memory (DAM) module. Each DAM module performs an adaptive combination of the response-level semantics captured from the encoder and the word-level semantics specifically selected for generating each word. Therefore, the responses contain more detailed and non-repetitive descriptions while maintaining the semantic accuracy. Furthermore, DAM is flexible to cooperate with existing visual dialogue encoders and adaptive to the encoder structures by constraining the information selection mode in DAM. We apply DAM to three typical encoders and verify the performance on the VisDial v1.0 dataset. Experimental results show that the proposed models achieve new state-of-the-art performance with high-quality responses. The code is available at https://github.com/JXZe/DAM.

Fact-based Visual Question Answering (FVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing FVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the final answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. In this paper, we depict an image by a multi-modal heterogeneous graph, which contains multiple layers of information corresponding to the visual, semantic and factual features. On top of the multi-layer graph representations, we propose a modality-aware heterogeneous graph convolutional network to capture evidence from different layers that is most relevant to the given question. Specifically, the intra-modal graph convolution selects evidence from each modality and cross-modal graph convolution aggregates relevant information across different graph layers. By stacking this process multiple times, our model performs iterative reasoning across three modalities and predicts the optimal answer by analyzing all question-oriented evidence. We achieve a new state-of-the-art performance on the FVQA task and demonstrate the effectiveness and interpretability of our model with extensive experiments.

Visual dialogue is a challenging task that needs to extract implicit information from both visual (image) and textual (dialogue history) contexts. Classical approaches pay more attention to the integration of the current question, vision knowledge and text knowledge, despising the heterogeneous semantic gaps between the cross-modal information. In the meantime, the concatenation operation has become de-facto standard to the cross-modal information fusion, which has a limited ability in information retrieval. In this paper, we propose a novel Knowledge-Bridge Graph Network (KBGN) model by using graph to bridge the cross-modal semantic relations between vision and text knowledge in fine granularity, as well as retrieving required knowledge via an adaptive information selection mode. Moreover, the reasoning clues for visual dialogue can be clearly drawn from intra-modal entities and inter-modal bridges. Experimental results on VisDial v1.0 and VisDial-Q datasets demonstrate that our model outperforms existing models with state-of-the-art results.

Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

Graph pattern matching has wide applications in social network analysis, such as identifying important communities, social roles and hidden behavior structures. Existing algorithms are mainly designed for single-pattern tasks, where patterns are processed sequentially and independently. However, many scenarios require multiple patterns to be processed as a batch, where single-pattern scheme will cost much redundant computation caused by matching similar sub-structures in the pattern set. Therefore, the sequential graph pattern matching scheme is not always the most efficient. This paper aims to propose a multiple pattern graph optimization algorithm for the subgraph isomorphism task. We comprehensively study the structural correlations among the multiple patterns and represent them by a compact tree-structured index. To support fast insertion and deletion of the pattern index, we present a dynamic updating algorithm to avoid index reconstruction from scratch. Based on the index, an efficient matching algorithm is proposed to answer multiple patterns in a heuristic scheduling order and avoid redundant computation. Extensive experiments on real and synthetic datasets prove that our solution is several times faster comparing with the state-of-the-art work.