Recent diffusion-based models have significantly improved inpainting quality. However, existing methods struggle with multi-task inpainting due to conflicting optimization objectives, and current datasets are typically limited to task-specific scenarios, hindering joint training. To address these challenges, we propose MagicPaint, a unified diffusion-based inpainting model that supports object addition, removal, and unconditional inpainting across both text and image modalities. MagicPaint semantically decouples operation types and target content by learnable tokens in MMToken Module, effectively reconciling conflicting optimization objectives and enabling robust multi-task, multi-modal inpainting. Besides, a novel inpainting paradigm named MagicMask, encodes operating intent directly into the mask and applies a mask loss for spatially precise supervision. In addition, existing inpainting datasets are insufficient for multi-task and multi-modal scenarios, limiting the capability of inpainting models. Thus, we further introduce a new dataset comprising 2.1M image tuples. It is dedicatedly designed to support diverse inpainting scenarios and significantly improves upon existing datasets, particularly in object removal. Through efforts from both model and data perspectives, MagicPaint enables users to operate anything—add, remove or inpaint content which is specified through either text or image modalities in a seamless and unified manner. Extensive experiments demonstrate that MagicPaint achieves state-of-the-art performance across three key tasks (i.e., text-guided addition, image-guided addition, and object removal) and produces outputs with superior visual consistency and contextual fidelity compared to existing methods.

Recent advances in image editing tools, particularly those used in content-aware retouching and object-level manipulation, have raised significant concerns regarding the authenticity of digital images. While many Image Manipulation Detection and Localization (IMDL) methods have been proposed, they often struggle with subtle forgeries, intricate boundary artifacts, and manipulations generated by unseen editing techniques. In this work, we propose a novel edge-aware framework that leverages the strong natural image priors of pre-trained inpainting models to harmonize manipulated regions. By guiding the inpainting process with generated edge-aware masks, our method reconstructs tampered areas using surrounding context, yielding perceptually coherent results. The pixel-wise residual between the original and reconstructed images reveals manipulation-sensitive inconsistencies—particularly around editing boundaries—thereby enabling accurate and generalizable detection and localization. Extensive experiments across multiple benchmarks demonstrate that our approach achieves state-of-the-art performance, especially in challenging scenarios involving realistic and finely retouched image forgeries.

Chenghao Yang, Yuning Zhang, Zhoufutu Wen, Tao Gong, Jiaheng Liu, Qi Chu, Nenghai Yu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

As speech translation (ST) systems become increasingly prevalent, understanding their vulnerabilities is crucial for ensuring robust and reliable communication. However, limited work has explored this issue in depth. This paper explores methods of compromising these systems through imperceptible audio manipulations. Specifically, we present two approaches: (1) adapting perturbation-based techniques used for automatic speech recognition (ASR) attacks to the ST context, making our work the first to apply this approach to ST, and (2) proposing a novel music generation-based method to guide targeted translation, while also conducting more practical over-the-air attacks in the physical world. Our experiments reveal that carefully crafted audio perturbations can mislead translation models to produce targeted, harmful outputs, while adversarial music achieve this goal more covertly, exploiting the natural imperceptibility of music. These attacks have proven effective across multiple languages and translation models, highlighting a systemic vulnerability in current ST architectures. Beyond immediate security concerns, our findings highlight broader challenges in the robustness and interpretability of neural speech systems.

Hair editing is a long-standing problem in computer vision that demands both fine-grained local control and intuitive user interactions across diverse modalities. Despite the remarkable progress of GANs and diffusion models, existing methods still lack a unified framework that simultaneously supports arbitrary interaction modes (e.g., text, sketch, mask, and reference image) while ensuring precise editing and faithful preservation of irrelevant attributes. In this work, we introduce a novel paradigm that reformulates hair editing as proxy-based hair transfer. Specifically, we leverage the dense and semantically disentangled latent space of StyleGAN for precise manipulation and exploit its feature space for disentangled attribute preservation, thereby decoupling the objectives of editing and preservation. Our framework unifies different modalities by converting editing conditions into distinct transfer proxies, whose features are seamlessly blended to achieve global or local edits. Beyond 2D, we extend our paradigm to 3D-aware settings by incorporating EG3D and PanoHead, where we propose a multi-view boosted hair feature localization strategy together with 3D-tailored proxy generation methods that exploit the inherent properties of 3D-aware generative models. Extensive experiments demonstrate that our method consistently outperforms prior approaches in editing effects, attribute preservation, visual naturalness, and multi-view consistency, while offering unprecedented support for multimodal and mixed-modal interactions.

With the rapid emergence of various generative models, generated images have increasingly become a prominent data medium on social platforms, making up a significantly higher proportion and providing fertile ground for steganography. However, research on steganography for generated images remains limited, and the distinctive attributes, especially the reproducibility of text-to-image (TTI) models, have not been effectively leveraged. In this paper, we propose GIANT (Generated Image Adversarial steganography based on Narrowed Targeting), a novel adversarial steganography framework for generated images that employs narrowed targeting to focus on embedding the secret message solely in the secure region and synchronizing the position to enhance the steganography security. GIANT achieves narrowed targeting by leveraging the reproducibility of TTI models and fusing two regions: (1) the minimal distortion region, which is localized by measuring steganographic distortion to evaluate the impact of modifications on the cover image distribution, and (2) the critical attention region, which is localized by using coarse-grained and fine-grained attention maps to evade steganalysis detection. Additionally, for positional synchronization of the secure region, the related prompts are transmitted alongside the stego image, allowing the receiver to reconstruct the cover image using a shared key and the provided prompt. Experimental results demonstrate that GIANT significantly improves security compared to conventional and adversarial steganographic methods designed for natural images, effectively countering state-of-the-art steganalyzers.

The rapid progress of generative AI has led to the emergence of new generative models, while existing detection methods struggle to keep pace with new model series and architectures, resulting in significant degradation in the detection performance. This highlights the urgent need for continuously updating AI-generated image detectors to adapt to new generators. To overcome low efficiency and catastrophic forgetting in detector updates, we propose LiteUpdate, a lightweight framework for updating AI-generated image detectors to unseen generative models. Unlike previous approaches that use randomly sampled training data, LiteUpdate employs a representative sample selection module that leverages image confidence and gradient-based discriminative features to precisely select boundary samples. This approach improves learning and detection accuracy on new distributions with limited generated images, significantly enhancing detector update efficiency. Additionally, LiteUpdate incorporates a model merging module that fuses weights from multiple fine-tuning trajectories, including pre-trained, representative, and random updates. This balances the adaptability to new generators and mitigates the catastrophic forgetting of previously learned knowledge. Experiments demonstrate that LiteUpdate substantially boosts detection performance in various detectors with high efficiency. Specifically, on AIDE, the average detection accuracy on Midjourney improved from 87.63% to 93.03%, a 6.16% relative increase. Meanwhile, to achieve comparable accuracy, LiteUpdate attains approximately 4× speedup over conventional random sample fine-tuning.

People are accustomed to sharing images on online social networks (OSNs), which is suited for covert communication. Robust steganography can survive JPEG recompression from OSN but sacrifices considerable security. This paper endeavors to achieve high-security robust steganography, an issue that plagues the current field. We define truncation-affected coefficients as unstable coefficients; they are major sources of error during recompression.We find that the unstable coefficients remain unstable under repeated recompression, which means both sides of the steganographic communication can naturally synchronize their positions. Therefore, we propose the unstable position synchronization based erasure code (UPSEC), where we replace the value of the unstable coefficient with a constant and set the distortion of the unstable coefficient to infinity when embedding. The receiver locates the unstable coefficients and sets their values to the same constant before extraction. By UPSEC, the unstable coefficient is effectively erased, so recompression errors no longer influence message extraction. Therefore, UPSEC achieves improved security without compromising robustness. Experimental results show that UPSEC can improve security by about +14% when compared to the state-of-the-art method against SRNet.

Previous research has demonstrated that a spatial domain image can provide side-information to its downsampled cover image, allowing a steganographer to embed a secret message on the cover image more securely by modulating the initial distortion. Importantly, the steganographer must possess the original image with a higher resolution than the cover image. In practical scenarios, however, the steganographer typically only has the cover image in which he wishes to embed the secret message; he does not have access to the real, higher-resolution image. To improve the security of steganography, we would like to estimate the side-information from the cover image. This paper proposes a spatial domain image steganography framework of side-information estimated with polarity adjustment strategy based on dual-path super-resolution reconstruction, in which double estimated side-information can be used to modulate the initial distortion. How to estimate more realistic high-resolution images and how to develop an effective modulation strategy are the central issues of our methods. We use double super-resolution networks to reconstruct high-resolution images for estimating side-information, and then propose a simple and effective strategy to modulate the initial distortion using dual-path estimated side-information. Experiments demonstrate that the security of dual-path side-information steganography can significantly outperform that of conventional distortion techniques.

The rise of deepfakes has significantly heightened concerns for privacy and the authenticity of digital media, bringing widespread attention to face forgery detection. Previous deepfake detection methods mostly depend on low-level textural features vulnerable to perturbations and fall short of detecting unseen forgery methods. In contrast, high-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. Motivated by this, we propose a detection method that utilizes high-level semantic features of faces to identify inconsistencies in temporal domain. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video classification network, initialized with a meta-functional face encoder for enriched facial representation. In this way, we can take advantage of both the powerful spatio-temporal model and the high-level semantic information of faces. Furthermore, to leverage easily accessible real face data and guide the model in focusing on spatio-temporal features, we design a Dynamic Video Self-Blending (DVSB) method to efficiently generate training samples with diverse spatio-temporal forgery traces using real facial videos. Based on this, we advance our framework with a two-stage training approach: The first stage employs a novel self-supervised contrastive learning, where we encourage the network to focus on forgery traces by impelling videos generated by the same forgery process to have similar representations. On the basis of the representation learned in the first stage, the second stage involves fine-tuning on face forgery detection dataset to build a deepfake detector. Extensive experiments validates that UniForensics outperforms existing face forgery detection methods in generalization ability and robustness. In particular, our method achieves 95.3% and 77.2% cross dataset AUC on the challenging Celeb-DFv2 and DFDC respectively. Code will be made publicly available.

Image steganography is an essential technique for concealing information by embedding secret information within images to make it undetectable. In recent years, with the rapid development and popularization of text-to-image generation models, many generated images have been disseminated through the Internet, thus making generated images ideal covers for steganography. Given that the distribution of generated images is more easily modeled than natural images, steganographic methods based on generated images exhibit higher security. Nevertheless, these methods typically require white-box access to the generative model, while contemporary popular generative models are black-box models. We observed that slight modifications in the input parameters of black-box image generative models result in subtle differences between generated images, offering new camouflage advantages for image steganography. Based on this observation, we propose an image steganography method based on the fluctuation of generative models. This approach leverages the fluctuation of image generative models, disguising stego images to appear as if they were generated by the parameter fluctuations of the generative model. Experimental results show that our proposed method outperforms baseline methods when facing steganalysis attacks, significantly enhancing steganographic security without compromising image quality.

With the development of deep learning, high-value and high-cost models have become valuable assets, and related intellectual property protection technologies have become a hot topic. However, existing model watermarking work in black-box scenarios originates mainly from training-based backdoor methods, which probably degrade primary task performance. To address this, we propose a branch backdoor-based model watermarking protocol named BranchWM to protect the intellectual property of the model. This protocol adopts a construction based on a message authentication scheme as the branch indicator, following a comparative analysis with other secure cryptographic primitives. We prove the lossless performance of the protocol by reduction. In addition, we analyze potential threats to the protocol and present a secure and feasible watermarking instantiation for language models. We further conduct empirical evaluations of the instantiated BranchWM, demonstrating its effectiveness and security for ownership verification.

Online social networks (OSNs) offer an abundant and freely available source of images, providing fertile ground for steganographic communication. However, the mandatory lossy operations applied by these platforms—primarily JPEG recompression— make robustness a pressing challenge. Existing robust steganographic methods focus on improving the embedding process, but inevitably compromise security. In this paper, we break this trade-off by proposing, for the first time, a robust cover screening method that enables successful message extraction after JPEG recompression, even when combined with non-robust steganographic methods. To ensure that the screened covers are compatible with arbitrary steganographic settings—including distortion functions, coding schemes, and messages—we introduce Robustness-Minimizing Modification (RMM), which simulates the worst-case impact of steganographic modifications on cover robustness. Images that remain unchanged under JPEG recompression after RMM are screened as robust covers. Our experiments reveal that such robust covers exist widely in both natural and generated images. Therefore, recent advances in generative modeling enable cost-effective and scalable expansion of candidate covers, addressing potential limitations of the screening method in practice. Our experiments also demonstrate that these screened covers can achieve 100% message extraction even with non-robust steganography at high embedding rates, while maintaining security comparable to other covers.

In domains like driving and healthcare, deep learning models often rely on large, diverse datasets that can inadvertently harbor backdoor attacks. In this paper, we propose ABDP, a novel post-processing defense method, to effectively remove backdoor contamination from datasets and generate clean models without relying on any pre-existing clean data. ABDP capitalizes on the intrinsic link between untargeted adversarial attacks and backdoor attacks to detect the presence of backdoor attacks within trained models and ascertain their target labels. Subsequently, it trains a clean model capable of recognizing all labels except the target label, thus treating poisoned data as in-distribution and clean data of the target label as out-of-distribution. This distinction enables the identification of backdoor poisoned data. Finally, ABDP applies unlearning techniques to effectively eradicate the backdoor from the model. Extensive experimental evaluations across diverse datasets and against multiple backdoor attack scenarios validate the robustness and state-of-the-art performance of our approach. By employing ABDP for data and model cleansing, the attack success rate of resulting models is reduced to 1% or less, while retaining approximately 70% or more clean data at a true positive rate of 0.01 false positive rate. Notably, ABDP exhibits no adverse impact when applied to purely clean datasets, owing to its ability to detect backdoor presence in models before cleansing. Thus, our proposed method achieves state-of-the-art performance in cleansing both backdoor-poisoned data and backdoor models. The code for ABDP will be made available upon publication of the paper.

Existing provably secure linguistic steganographic methods typically rely on white-box extraction, which necessitates access to large language models. This requirement is impractical in environments with limited resources. To tackle this issue, we proposeDisreo, a provably secure linguistic steganography based ondistributionreorganization, which extracts the messages without accessing the underlying language model. This is achieved through token position randomization and output probability reorganization for message embedding. Moreover, secret message extraction requires only the synchronization of token positions used during embedding, making it both feasible and fast for devices with constrained computational capabilities. Theoretically, the security ofDisreocan be reduced to the security of the encryption algorithm we employ, and our experimental analyses confirm thatDisreomaintains distribution consistency between stego and cover texts in expectations. In practice,Disreoachieves an average extraction time of 0.015 seconds for 5 bits of secret messages from 100 tokens, with a 100% extraction accuracy. By transitioning from white-box extraction to more practical no-box extraction scenarios,Disreobroadens the scope of steganography applications.

Analyzing the behavior of cryptographic functions in stripped binaries is a challenging but essential task, which is crucial in software security fields such as malware analysis and legacy code inspection. However, the inherent high logical complexity of cryptographic algorithms makes their analysis more difficult than that of ordinary code, and the general absence of symbolic information in binaries exacerbates this challenge. Existing methods for cryptographic algorithm identification frequently rely on data or structural pattern matching, which limits their generality and effectiveness while requiring substantial manual effort. In response to these challenges, we present F igure o ut the C ryptographic functions (FoC), a novel framework that leverages Large Language Models (LLMs) to identify and analyze cryptographic functions in stripped binaries. In FoC, we first build an LLM-based generative model ( FoC-BinLLM ) to summarize the semantics of cryptographic functions in natural language form, which is intuitively readable to analysts. Subsequently, based on the semantic insights provided by FoC-BinLLM, we further develop a binary code similarity detection model ( FoC-Sim ), which allows analysts to effectively retrieve similar implementations of unknown cryptographic functions from a library of known cryptographic functions. The predictions of generative model like FoC-BinLLM are inherently difficult to reflect minor alterations in binary code, such as those introduced by vulnerability patches. In contrast, the change-sensitive representations generated by FoC-Sim compensate for the shortcomings to some extent. To support the development and evaluation of these models, and to facilitate further research in this domain, we also construct a comprehensive cryptographic binary dataset and introduce an automatic method to create semantic labels for extensive binary functions. Our evaluation results are promising. FoC-BinLLM outperforms ChatGPT by 14.61% on the ROUGE-L score, demonstrating superior capability in summarizing the semantics of cryptographic functions. FoC-Sim also surpasses previous best methods with a 52% higher Recall@1 in retrieving similar cryptographic functions. Beyond these metrics, our method has proven its practical utility in real-world scenarios, including cryptographic-related virus analysis and 1-day vulnerability detection.

In 3D human action recognition, limited supervised data makes it challenging to fully tap into the modeling potential of powerful networks such as transformers. As a result, researchers have been actively investigating effective self-supervised pre-training strategies. For example, MAMP shows that instead of following the prevalent masked joint reconstruction, explicit masked motion reconstruction is key to the success of learning effective feature representation for 3D action recognition. However, we find that if we make a simple and effective change to the reconstructed target of masked joint reconstruction, masked joint reconstruction can achieve the same results as masked motion reconstruction. The devil is in the special characteristic of 3D skeleton data and the normalization process of training targets. We need to dig for all effective information of targets during normalization. Besides, considering that mask data reconstruction focuses more on learning local relations in input data for fulfilling the reconstruction task, instead of modeling the relation among samples, we further employ contrastive learning to learn more discriminative 3D action representations. We show that contrastive learning can consistently boost the performance of model pre-trained by masked joint prediction under various settings, especially in the semi-supervised setting that has a very limited number of labeled samples. Extensive experiments on NTU-60, NTU-120, and PKU-MMD datasets show that the proposed pre-training strategy achieves state-of-the-art results without bells and whistles.

Open-vocabulary semantic segmentation (OVSS) aims to segment images of arbitrary categories specified by class labels. While previous approaches relied on extensive image-text pairs or dense semantic annotations, recent training-free methods attempted to overcome these limitations by constructing semantic prototypes in the construction stage and image-to-image matching (i.e., prototype matching) during testing. However, these methods often struggle to effectively capture the visual characteristics of categories and fail to utilize local features during prototype matching. To deal with these problems, we propose a novel training-free framework for OVSS that constructs diverse prototypes and performs fine-grained sub-region matching. Specifically, our method leverages Large Language Models (LLMs) to guide support image generation by descriptions of different attributes of categories and employs coarse-fine clustering to obtain diverse and robust part-level prototypes in the construction stage. During testing, we propose a sub-region matching method, which assigns part-level prototypes to sub-regions utilizing optimal transport, to fully utilize local image features among part-level prototypes. Extensive experiments demonstrate the effectiveness of our method and show that our method achieves state-of-the-art performance, outperforming previous methods across five datasets.

Li Hu, Guoqiang Chen, Xiuwei Shang, Shaoyin Cheng, Benlong Wu, LiGangyang LiGangyang, Xu Zhu, Weiming Zhang, Nenghai Yu. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Text-to-image models have shown remarkable capabilities in generating high-quality images from natural language descriptions. However, these models are highly vulnerable to adversarial prompts, which can bypass safety measures and produce harmful content. Despite various defensive strategies, achieving robustness against attacks while maintaining practical utility in real-world applications remains a significant challenge. To address this issue, we first conduct an empirical study of the text encoder in the Stable Diffusion (SD) model, which is a widely used and representative text-to-image model. Our findings reveal that the [EOS] token acts as a semantic aggregator, exhibiting distinct distributional patterns between benign and adversarial prompts in its embedding space. Building on this insight, we introduce SafeGuider, a two-step framework designed for robust safety control without compromising generation quality. SafeGuider combines an embedding-level recognition model with a safety-aware feature erasure beam search algorithm. This integration enables the framework to maintain high-quality image generation for benign prompts while ensuring robust defense against both in-domain and out-of-domain attacks. SafeGuider demonstrates exceptional effectiveness in minimizing attack success rates, achieving a maximum rate of only 5.48% across various attack scenarios. Moreover, instead of refusing to generate or producing black images for unsafe prompts, SafeGuider generates safe and meaningful images, enhancing its practical utility. In addition, SafeGuider is not limited to the SD model and can be effectively applied to other text-to-image models, such as the Flux model, demonstrating its versatility and adaptability across different architectures. We hope that SafeGuider can shed some light on the practical deployment of secure text-to-image systems.

Recent advancements in in-context segmentation generalists have demonstrated significant success in performing various image segmentation tasks using a limited number of labeled example images. However, real-world applications present challenges due to the variability of support examples, which often exhibit quality issues resulting from various sources and inaccurate labeling. How to extract more robust representations from these examples has always been one of the goals of in-context visual learning. In response, we propose UNICL-SAM, to better model the example distribution and extract robust representations to help in-context segmentation. We incorporate an uncertainty probabilistic module to quantify each example’s reliability during both the training and testing phases. Utilizing this uncertainty estimation, we introduce an uncertainty-guided graph augmentation and feature refinement strategy, aimed at mitigating the impact of high-uncertainty regions to enhance the learning of robust representations. Subsequently, we construct prototypes for each example by aggregating part information, thereby creating reliable in-context instruction that effectively represents fine-grained local semantics. This approach serves as a valuable complement to traditional global pooling features. Experimental results demonstrate the effectiveness of the proposed framework, underscoring its potential for real-world applications.

Vertical federated learning (VFL) is a collaborative learning scheme where clients share some overlapping samples but have different feature spaces. Existing VFL schemes are restricted in model performance and deployment feasibility due to the scarcity of overlapping labeled samples and high communication costs. To tackle these issues, we propose a practical VFL scheme Semi-VFL using stacked generalization, which can effectively improve model performance with limited aligned labeled samples and only two communication times. A built-in local semi-supervised learning strategy FewMatch with model-level consistency for few-label VFL setting is designed in our scheme. Extensive experiments indicate the superiority of Semi-VFL in both image and tabular datasets. Specifically, Semi-VFL can achieve accuracy improvement by more than 10.9% and communication cost reduction by more than 380× over the state-of-the-art few-shot VFL scheme on CIFAR-10 with 128 aligned samples.

Knowledge Distillation (KD) aims to enhance the performance of student models by transferring knowledge from teacher models. While reaping the benefits of KD, the intellectual property risks associated with it cannot be ignored. Even if models are released without training data or provided as a service, potential adversaries can still clone the target model using KD. To mitigate the risks, some researchers propose training the anti-KD model that cannot teach student models. However, we find existing methods cannot defend against representation-based KD. To address the knowledge leakage from representations, we introduce Similarity Disruption (SD). SD increases the distance between the representation similarity matrices of our anti-KD model and the normal model, thereby reducing the effective information in the representation space. Extensive experiments demonstrate the proposed method can effectively defend against representation-based KD.

Large language models (LLMs) are typically pre-trained on massive-scale data and then fine-tuned on specialized datasets for downstream applications. Determining whether a model has been fine-tuned on a particular dataset is crucial for dataset owners to protect their copyrights. However, there is a lack of effective methods for safeguarding datasets used in LLM fine-tuning. This paper proposes a robust watermarking scheme that inserts tokens representing watermark information into datasets without altering their original semantics. When these watermarked datasets are used for fine-tuning, the watermark can be reliably extracted from the model’s generated outputs. Experiments show that models fine-tuned on our watermarked dataset successfully reveal the watermark in their outputs while maintaining performance comparable to those trained on non-watermarked datasets. When the fine-tuning dataset is entirely embedded with our watermark, the extraction success rate from the generated outputs during inference reaches up to 97.4%.

Combination usage of LiDAR and RGB cameras for gait recognition can achieve cross space recognition and privacy protection. In addition, the widespread application of LiDAR cameras with 3D geometry information and the large amount of RGB gaits has led to the demand for cross-modality gait recognition on LiDAR and RGB modalities. To address the challenge of cross-modality recognition, we proposed a novel cross-modality gait recognition paradigm called CMGait. The key innovations include a novel projection method for transforming LiDAR point clouds into depth maps, feature alignment modules, Transformer-based identity encoders, and an embedding distance fusion method with similarity matrices based contrastive learning. Experimental results showcase state-of-the-art performance with Rank-1 accuracy of 62.8% and 66.1% for different directions in cross-modality gait recognition. Ablation experiments validate the effectiveness of the proposed methods, highlighting advancements in feature alignment and modality fusion techniques.

The widespread adoption of face recognition has raised privacy concerns regarding the collection and use of facial data. To address this, researchers have explored "unlearnable examples" by adding imperceptible perturbations during model training to prevent the model from learning target features. However, current methods are inefficient and cannot guarantee transferability and robustness at the same time, causing impracticality in the real world. To remedy it, we introduce Side-information Guided Generative Unlearnable Examples (Segue). Using a once-trained multiple-used model to generate perturbations, Segue avoids the time-consuming gradient-based approach. To improve transferability, we introduce side information such as true or pseudo labels, which are inherently consistent across different scenarios. For robustness enhancement, a distortion layer is integrated into the training pipeline. Experiments show Segue is 1000× faster than previous methods, transferable across datasets and models, and resistant to JPEG compression, adversarial training, and standard augmentations.

AI-generated content (AIGC) enables efficient visual creation but raises copyright and authenticity risks. As a common technique for integrity verification and source tracing, digital image watermarking is regarded as a potential solution to above issues. However, the widespread adoption and advancing capabilities of generative image editing tools have amplified malicious tampering risks, while simultaneously posing new challenges to passive tampering detection and watermark robustness. To address these challenges, this paper proposes a Tamper-Aware Generative image WaterMarking method named TAG-WM. The proposed method comprises four key modules: a dual-mark joint sampling (DMJS) algorithm for embedding copyright and localization watermarks into the latent space while preserving generative quality, the watermark latent reconstruction (WLR) utilizing reversed DMJS, a dense variation region detector (DVRD) leveraging diffusion inversion sensitivity to identify tampered areas via statistical deviation analysis, and the tamper-aware decoding (TAD) guided by localization results. The experimental results demonstrate that TAG-WM achieves state-of-the-art performance in both tampering robustness and localization capability even under distortion, while preserving lossless generation quality and maintaining a watermark capacity of 256 bits. The code is available at: https://github.com/Suchenl/TAG-WM.

Recent advancements in unified image generation models, such as OmniGen, have enabled the handling of diverse image generation and editing tasks within a single framework, accepting multimodal, interleaved texts and images in free form. This unified architecture eliminates the need for text encoders, greatly reducing model complexity and standardizing various image generation and editing tasks, making it more user-friendly. However, we found that it suffers from text instruction neglect, especially when the text instruction contains multiple sub-instructions. To explore this issue, we performed a perturbation analysis on the input to identify critical steps and layers. By examining the cross-attention maps of these key steps, we observed significant conflicts between neglected sub-instructions and the activations of the input image. In response, we propose Self-Adaptive Attention Scaling (SaaS), a method that leverages the consistency of cross-attention between adjacent timesteps to dynamically scale the attention activation for each sub-instruction. Our SaaS enhances instruction-following fidelity without requiring additional training or test-time optimization. Experimental results on instruction-based image editing and visual conditional image generation validate the effectiveness of our SaaS, showing superior instruction-following fidelity over existing methods. The code is available https://github.com/zhouchao-ops/SaaS.

Most of the existing rotation detection models focus on solving problems such as feature misalignment and boundary discontinuity, but ignore the use of contextual information in remote sensing images. However, context information plays a vital role in the accurate detection of small instance targets. Especially when the receptive field of the model is limited, it is very easy to cause deviations in the detection results. Therefore, we propose a Remote Sensing Target Detector with Multi-scale Attention Mechanism Network(MAMNet) that gradually enhances the features in the region of interest by utilizing the spatial, local, and global information of the image input features, thereby supplementing the attention mechanism at multiple scales, better identifying targets, and improving the detection performance of the model. Our model was experimented on DOTAv1.0, DOTAv1.5, HRSC2016 and has achieved the state of the art on DOTAv1.0.

Watermarking photographs is a crucial tool for safeguarding copyrights and can serve as a more aesthetically pleasing alternative to QR codes. In recent years, watermarking methods based on deep learning have proved superior robustness against complex physical distortions than traditional watermarking methods. However, they have some limitations that render them less effective in practice. For instance, current solutions necessitate physical photographs to be rectangular for accurate localization, can’t handle physical bending or folding, and require the hidden area to be completely captured at a close distance and small angle. To overcome these challenges, we propose a novel deep watermarking framework dubbed Aparecium. Specifically, we preprocess secrets (i.e., watermarks) into a visible pattern and then embed it into the cover image invisibly, which is symmetrical to the final decoding-then-extracting process. To capture the watermarked region from complex physical scenarios, edge distortion is also introduced. Finally, we adopt a three-stage training strategy for training convergence. Extensive experiments demonstrate that Aparecium is not only robust against different digital distortions, but also can resist different physical distortions, such as screen-shooting and printing-shooting, even in severe cases including different shapes, curvature, folding, incompleteness, long distances, and big angles while maintaining high visual quality. Furthermore, some ablation studies are also conducted to verify our design.

Deep image watermarking embeds identification data into images to facilitate source tracking. However, existing schemes are primarily designed for single-stage transmission scenarios, and in practical multi-stage distribution requirements, current methods degrade image quality and reduce watermark extraction accuracy. In this paper, we introduces WaterUp, a deep watermark updating framework. WaterUp automatically updates watermark information as the image is transmitted, preserving image quality while accurately recording the transmission path for traceability. The core of WaterUp is a flow-based encoder-decoder (FED), which utilizes a forward and backward network to enable efficient watermark updating with minimal computational and storage demands. Experimental results show that WaterUp outperforms state-of-the-art methods, maintaining high visual quality with a PSNR exceeding 38 dB across multiple transmissions.

Deep neural networks have found widespread application in critical fields but remain vulnerable to adversarial attacks. Existing detection methods aim to achieve defense without modifying the model, but they generally struggle with generalization to unseen attacks. To address this limitation, we investigate the underlying principles of max-loss and min-distance adversarial attacks and uncover a strong positive correlation between perturbation magnitude, prediction confidence, and the distance to the decision boundary. Building on this insight, we introduce Adversarial Detection via Adversarial Sensitivity (ADAS), a novel approach that detects adversarial attacks by analyzing the sensitivity of a model's predictions to perturbation magnitude. ADAS estimates the distance to the decision boundary through sensitivity analysis by simulating adversarial attacks on input samples, identifying anomalies indicative of adversarial manipulation. Extensive experiments demonstrate the robustness and generalizability of ADAS across diverse and previously unseen adversarial attack scenarios, establishing its efficacy as a versatile and reliable detection framework.

The rapid advancement of speech generation models has heightened privacy and security concerns related to voice cloning (VC). Recent studies have investigated disrupting unauthorized voice cloning by introducing adversarial perturbations. However, determined attackers can mitigate these protective perturbations and successfully execute VC. In this study, we conduct the first systematic evaluation of these protective perturbations against VC under realistic threat models that include perturbation purification. Our findings reveal that while existing purification methods can neutralize a considerable portion of the protective perturbations, they still lead to distortions in the feature space of VC models, which degrades the performance of VC. From this perspective, we propose a novel two-stage purification method: (1) Purify the perturbed speech; (2) Refine it using phoneme guidance to align it with the clean speech distribution. Experimental results demonstrate that our method outperforms state-of-the-art purification methods in disrupting VC defenses. Our study reveals the limitations of adversarial perturbation-based VC defenses and underscores the urgent need for more robust solutions to mitigate the security and privacy risks posed by VC. The code and audio samples are available at https://de-antifake.github.io.

In real applications, person re-identification (ReID) expects to retrieve the target person at any time, including both daytime and nighttime, ranging from short-term to long-term. However, existing ReID tasks and datasets cannot meet this requirement, as they are constrained by available time and only provide training and evaluation for specific scenarios. Therefore, we investigate a new task called Anytime Person Re-identification (AT-ReID), which aims to achieve effective retrieval in multiple scenarios based on variations in time. To address the AT-ReID problem, we collect the first large-scale dataset, AT-USTC, which contains 135k images of individuals wearing multiple clothes captured by RGB and IR cameras. Our data collection spans over an entire year and 270 volunteers were photographed on average 29.1 times across different dates or scenes, 4-15 times more than current datasets, providing conditions for follow-up investigations in AT-ReID. Further, to tackle the new challenge of multi-scenario retrieval, we propose a unified model named Uni-AT, which comprises a multi-scenario ReID (MS-ReID) framework for scenario-specific features learning, a Mixture-of-Attribute-Experts (MoAE) module to alleviate inter-scenario interference, and a Hierarchical Dynamic Weighting (HDW) strategy to ensure balanced training across all scenarios. Extensive experiments show that our model leads to satisfactory results and exhibits excellent generalization to all scenarios.

Binary analysis is crucial for software security, offering insights into compiled programs without source code. As large language models (LLMs) excel in language tasks, their potential for complex decoding binary data structures is growing. However, the lack of standardized benchmarks hinders their evaluation and progress in this domain. To bridge this gap, we introduce BinMetric, a first comprehensive benchmark designed specifically to evaluate LLMs performance on binary analysis tasks. BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks, including decompilation, code summarization, etc., which reflect actual reverse engineering scenarios. Our empirical study on this benchmark investigates various state-of-the-art LLMs, revealing their strengths and limitations. The findings indicate that while LLMs show strong potential, challenges still exist, particularly in the areas of precise binary lifting and assembly synthesis. In summary, BinMetric makes a significant step forward in measuring binary analysis capabilities of LLMs, establishing a new benchmark leaderboard, and our study offers valuable insights for advancing LLMs in software security.

Decompilation can convert binary programs into clear C-style pseudocode, which is of great value in a wide range of security applications. Existing research primarily focuses on recovering symbolic information in pseudocode, such as function names, variable names, and data types, but neglecting structural information. We observe that even when symbolic information is fully preserved, severe and complex structure distortions remain in the pseudocode, greatly impairing code readability and comprehension. In this work, we first systematically investigate structure distortions in decompiled pseudocode, revealing their variation patterns through quantitative analysis. Using open coding, we derive a taxonomy comprising six top-level categories of structure distortions. Building upon this taxonomy, we propose PseudoFix, a novel framework that combines large language models (LLMs) with retrieval-based in-context learning. PseudoFix employs semantic retrieval to select the most relevant few-shot examples that provide structure distortion knowledge, and combines this with the well-structured coding patterns learned by LLMs from vast source code repositories, to efficiently refactor distorted pseudocode. Comprehensive evaluations demonstrate that PseudoFix significantly improves pseudocode readability, achieving up to a 34% reduction in Halstead Complexity Effort and a 105% increase in BLEU-4 score. Notably, it significantly outperforms state-of-the-art approaches in both temporary variable elimination and goto statement removal tasks. Additionally, human evaluations yield consistently positive feedback from users across readability, consistency, and reasonability.

With the advancement of face manipulation technology, forgery images in multi-face scenarios are gradually becoming a more complex and realistic challenge. Despite this, detection and localization methods for such multi-face manipulations remain underdeveloped. Traditional manipulation localization methods either indirectly derive detection results from localization masks, resulting in limited detection performance, or employ a naive two-branch structure to simultaneously obtain detection and localization results, which cannot effectively benefit the localization capability due to limited interaction between the two tasks. This paper proposes a new framework, namely MoNFAP, specifically tailored for multi-face manipulation detection and localization. The MoNFAP primarily introduces two novel modules: the Forgery-aware Unified Predictor (FUP) Module and the Mixture-of-Noises Module (MNM). The proposed FUP integrates detection and localization tasks using a token learning strategy and multiple forgery-aware transformers, which facilitates the use of classification information to enhance localization capability. Furthermore, to mitigate the interference from general semantic object information, we propose the MNM that leverages multiple noise extractors based on the mixture of experts concept. This allows the MNM to learn semantic-agnostic forgery features from general RGB features, further boosting the performance of our proposed framework. Finally, we establish a comprehensive benchmark for multi-face detection and localization, and the proposed MoNFAP achieves significant performance. The code is available: https://github.com/miaoct/MoNFAP.

Advances in automatic graphical user interface (GUI) agents have brought significant privacy and security challenges, especially cloud-based solutions that may leak sensitive data and be vulnerable to man-in-the-middle attacks. To address these issues, we propose MMPro, a novel GUI agent framework that adopts a separated perception-thinking-execution architecture. The perception module and the execution module process inputs locally to generate outputs to ensure security, and the thinking module operates on abstract representations to ensure the effectiveness of the GUI agent.By modularizing each stage while introducing a hybrid description language (HDL), MMPro transforms screen images into abstract structured representations, minimizing the risk of sensitive information leakage. Experimental results on the OSWorld benchmark show that MMPro outperforms existing GUI agent methods while ensuring strong privacy protection and real-time interaction capabilities. This work presents a pioneering approach to developing efficient, privacy-conscious, and explainable automated GUI agents.

The widespread adoption of Low-Rank Adaptation (LoRA) modules in parameter-efficient fine-tuning has revolutionized the deployment of large-scale deep neural networks. However, the intellectual property protection of LoRA modules remains a critical challenge. White-box watermarking is a more effective solution than black-box watermarking in the multi-bit verification scenario of protecting and tracing intellectual property. However, existing white-box watermarking methods for LoRA lack both flexible multi-bit capacity and merging robustness, leaving LoRA modules vulnerable to unauthorized use and redistribution. In this paper, we propose a novel merging-resistant watermarking method for LoRA modules. Our method embeds watermarks into the increment matrix generated during LoRA merging and decomposes the watermark-induced modifications into LoRA's standard matrices, achieving reliable watermark extraction and preserving LoRA's efficiency. Specifically, we adopt quantization index modulation to embed watermarks in the low-frequency components of selected increment matrix weights. Extensive experiments demonstrate the effectiveness, imperceptibility, and robustness of our method, making it a practical solution for safeguarding LoRA modules in real-world applications. This work responds to the limited attention given to intellectual property protection for LoRA, contributing to the secure and sustainable development of deep learning technologies.

A striking proficiency of diffusion models in producing and manipulating images with an unprecedented level of realism has unquestionably elicited concerns. Many methods have been proposed to detect generated images. In particular, recent studies reveal that autoencoder reconstruction error can serve as an effective indicator for distinguishing authentic and synthetic images, since most generative models adopt analogous encoder-decoder operation. However, the reliance on a single autoencoder reconstruction error provides only limited information, which is insufficient for comprehensively capturing discriminative features, resulting in restricted generalization performance. In this paper, we propose Multiple Reconstruction Contrastive Learning (MRCL), which leverages multiple reconstruction residuals to enhance the generalizability of generated image detection. Specifically, MRCL applies Dinov2-ViT with LoRA fine-tuning to extract fine-grained feature representations of origin images and their multiple VAE reconstructions. In addition, a Residual Dense Fusion module is designed to effectively combine multiple VAE reconstruction residuals. Further, a contrastive learning strategy is adopted to guide the distance of origin images and VAE reconstruction representations. Extensive experimental results demonstrate the superior generalization performance of the proposed MRCL.

Meng Tong, Kejiang Chen, Xiaojian Yuan, Jiayang Liu, Weiming Zhang, Nenghai Yu, Jie Zhang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Generative steganography has emerged as an active research area, yet its practical system is constrained by the inherent secret payload limitation caused by low entropy in generating stego texts. This payload limitation necessitates the use of lengthy stego texts or frequent transmissions, which increases the risk of suspicion by adversaries. Previous studies have mainly focused on payload enhancement through optimized entropy utilization while overlooking the crucial role of secret message processing. To address this gap, we propose StegoZip, a framework that leverages large language models to optimize secret message processing. StegoZip consists of two core components: semantic redundancy pruning and index-based compression coding. The former dynamically prunes the secret message to extract a low-semantic representation, whereas the latter further compresses it into compact binary codes. When integrated with state-of-the-art steganographic methods under lossless decoding, StegoZip achieves 2.5$\times$ the payload of the baselines while maintaining comparable processing time in practice. This enhanced payload significantly improves covertness by mitigating the risks associated with frequent transmissions while maintaining provable content security.

Recent provably secure linguistic steganography (PSLS) methods rely on mainstream autoregressive language models (ARMs) to address historically challenging tasks, that is, to disguise covert communication as ``innocuous'' natural language communication. However, due to the characteristic of sequential generation of ARMs, the stegotext generated by ARM-based PSLS methods will produce serious error propagation once it changes, making existing methods unavailable under an active tampering attack. To address this, we propose a robust, provably secure linguistic steganography with diffusion language models (DLMs). Unlike ARMs, DLMs can generate text in a partially parallel manner, allowing us to find robust positions for steganographic embedding that can be combined with error-correcting codes. Furthermore, we introduce error correction strategies, including pseudo-random error correction and neighborhood search correction, during steganographic extraction. Theoretical proof and experimental results demonstrate that our method is secure and robust. It can resist token ambiguity in stegotext segmentation and, to some extent, withstand token-level attacks of insertion, deletion, and substitution.

Vector databases support machine learning tasks using Approximate Nearest Neighbour (ANN) query functionality, making them highly valuable digital assets. However, they also face security threats like unauthorized replication. By embedding stealth information, watermarking technology can be used for ownership authentication. This paper introduces a watermarking scheme specifically designed for vector databases. The scheme consists of four steps: generating identifiers, grouping, cryptographic mapping, and modification. Since watermark embedding requires modification of certain vectors, it may negatively affect the ANN query results. Further investigation reveals that in the widely used Hierarchical Navigable Small World (HNSW) indexing structure for vector databases, heuristic edge selection and pruning strategies result in some vectors having fewer edges or even none at all. These vectors exhibit significantly lower query frequencies than others, which means that modifying these vectors incurs less impact on query results. Based on this observation, we propose the Transparent Vector Priority (TVP) watermarking scheme, which prioritizes embedding the watermark in these low-query-frequency “transparent” vectors to minimize the impact of watermark embedding on query results. Experimental results show that compared to the current most effective and relevant watermarking schemes, the TVP scheme can significantly reduce the number of missed and false queries by approximately 75\%.

Existing video steganography methods primarily embed secret information by modifying video content in the spatial or compressed domains. However, such methods are prone to distortion drift and are easily detected by steganalysis. Generative steganography, which avoids direct modification of the cover data, offers a promising alternative. Despite recent advances, most generative steganography studies focus on images and are difficult to extend to videos because of compression-induced distortions and the unique architecture of video generation models. To address these challenges, we propose LD-RoViS, a training-free and robust video steganography framework for the deterministic latent diffusion model. By modulating implicit conditional parameters during the diffusion process, LD-RoViS constructs a dedicated steganographic channel. Additionally, we introduce a novel multi-mask mechanism to mitigate errors caused by video compression and post-processing. The experimental results demonstrate that LD-RoViS can embed approximately 12,000 bits of data into a 5-second video with an extraction accuracy exceeding 99\%. Our implementation is available at https://github.com/xiangkun1999/LD-RoViS.

Diffusion models have advanced rapidly in recent years, producing high-fidelity images while raising concerns about intellectual property protection and the misuse of generative AI. Image watermarking for diffusion models, particularly Noise-as-Watermark (NaW) methods, encode watermark as specific standard Gaussian noise vector for image generation, embedding the infomation seamlessly while maintaining image quality. For detection, the generation process is inverted to recover the initial noise vector containing the watermark before extraction. However, existing NaW methods struggle to balance watermark robustness with generation diversity. Some methods achieve strong robustness by heavily constraining initial noise sampling, which degrades user experience, while others preserve diversity but prove too fragile for real-world deployment. To address this issue, we propose T2SMark, a two-stage watermarking scheme based on Tail-Truncated Sampling (TTS). Unlike prior methods that simply map bits to positive or negative values, TTS enhances robustness by embedding bits exclusively in the reliable tail regions while randomly sampling the central zone to preserve the latent distribution. Our two-stage framework then ensures sampling diversity by integrating a randomly generated session key into both encryption pipelines. We evaluate T2SMark on diffusion models with both U-Net and DiT backbones. Extensive experiments show that it achieves an optimal balance between robustness and diversity.

Hidden wireless cameras pose significant privacy threats, necessitating effective detection and localization methods. However, existing localization solutions often require impractical activity spaces, expensive specialized devices, or pre-collected training data, limiting their practical deployment. To address these limitations, we introduce CamLopa, a training-free wireless camera localization framework that operates with minimal activity space constraints using low-cost, commercial-off-the-shelf (COTS) devices. CamLopa can achieve detection and localization in just 45 seconds of user activities with a Raspberry Pi board. During this short period, it analyzes the causal relationship between wireless traffic and user movement to detect the presence of a hidden camera. Upon detection, CamLopa utilizes a novel azimuth localization model based on wireless signal propagation path analysis for localization. This model leverages the time ratio of user paths crossing the First Fresnel Zone (FFZ) to determine the camera's azimuth angle. Subsequently, CamLopa refines the localization by identifying the camera's quadrant. We evaluate CamLopa across various devices and environments, demonstrating its effectiveness with a 95.37% detection accuracy for snooping cameras and an average localization error of 17.23°, under the significantly reduced activity space requirements and without the need for training. Our code and demo are available at https://github.com/CamLoPA/CamLoPA-Code.

In recent years, multi-hop Quantum Key Distribution (QKD) network has been proven as a promising solution through rigorous practices to provide end-to-end key exchange service for arbitrary communication parties. However, existing decentralized solutions still face critical challenges including consistency and fairness that stem from storable nature of quantum key material. Thus, in this paper, we first devise a Key Management and Service (KM&S) framework for decentralized multi-hop QKD networks, which provides functional decoupling and pipeline processing to guarantee flexibility and compatibility for practical implementation. After that, to address consistency and fairness challenges during end-to-end key exchange service, we focus on two aspects including local key management and end-to-end congestion control, and respectively propose an elastic key supply rate control scheme named AUTO and a Capacity Probing-driven Backpressure Flow Control (CP-BFC) scheme. Furthermore, we construct an experiment platform equipped with realistic QKD devices based on China metropolitan QKD network topology to implement the proposed framework and schemes, and conduct extensive experiments compared to representative schemes in existing studies. The experimental results show that AUTO&CP-BFC significantly outperforms representative schemes in terms of consistency and fairness.

Face-swapping techniques have advanced rapidly with the evolution of deep learning, leading to widespread use and growing concerns about potential misuse, especially in cases of fraud. While many efforts have focused on detecting swapped face images or videos, these methods are insufficient for tracing the malicious users behind fraudulent activities. Intrusive watermark-based approaches also fail to trace unmarked identities, limiting their practical utility. To address these challenges, we introduce FaceTracer, the first non-intrusive framework specifically designed to trace the identity of the source person from swapped face images or videos. Specifically, FaceTracer leverages a disentanglement module that effectively suppresses identity information related to the target person while isolating the identity features of the source person. This allows us to extract robust identity information that can directly link the swapped face back to the original individual, aiding in uncovering the actors behind fraudulent activities. Extensive experiments demonstrate FaceTracer's effectiveness across various face-swapping techniques, successfully identifying the source person in swapped content and enabling the tracing of malicious actors involved in fraudulent activities. Additionally, FaceTracer shows strong transferability to unseen face-swapping methods including commercial applications and robustness against transmission distortions and adaptive attacks.

Visible watermarking serves as a crucial security mechanism for safeguarding the copyright of digital images. Recent advancements, however, have shown that deep neural networks can effectively remove these watermarks without altering the underlying host image, posing a substantial risk to copyright protection . Motivated by the susceptibility of neural networks to adversarial perturbations, various adversarial visible watermarking techniques have been introduced. Nonetheless, these approaches often overlook the need for image reversibility, which is vital for authorized sharing while maintaining privacy. To address this issue, we propose R eversible A dversarial V isible W atermarking (RAVW), which uses Gradient-weighted Class Activation Mapping (Grad-CAM) to pinpoint the important regions in the host image that are optimal for watermark embedding . It then employs an end-to-end generative model to create reversible adversarial visible watermarks within these regions, effectively counteracting watermark removal networks. Additionally, authorized users can eliminate the visible watermark via a dedicated restoration module. Comprehensive experimental evaluations confirm the robustness of our method in preserving visible watermarks and its effectiveness against watermark removal networks.

Malicious image manipulation poses societal risks, increasing the importance of effective image manipulation detection methods. Recent approaches in image manipulation detection have largely been driven by fully supervised approaches, which require labor-intensive pixel-level annotations. Thus, it is essential to explore weakly supervised image manipulation localization methods that only require image-level binary labels for training. However, existing weakly supervised image manipulation methods overlook the importance of edge information for accurate localization, leading to suboptimal localization performance. To address this, we propose a Context-Aware Boundary Localization (CABL) module to aggregate boundary features and learn context-inconsistency for localizing manipulated areas. Furthermore, by leveraging Class Activation Mapping (CAM) and Segment Anything Model (SAM), we introduce the CAM-Guided SAM Refinement (CGSR) module to generate more accurate manipulation localization maps. By integrating two modules, we present a novel weakly supervised framework based on a dual-branch Transformer-CNN architecture. Our method achieves outstanding localization performance across multiple datasets.

Quantum communication networks enable advanced quantum applications through remote entanglement distribution among source-destination pairs. Despite efforts to optimize entanglement distribution, fairness in multi-request scenarios has been neglected, potentially causing issues like “request starvation”. To address such issue, this paper concentrates on the unique properties of entangled systems and introduces a process-oriented fairness metric, i.e., expected throughput, departing from conventional approaches used in classical networks. Furthermore, we propose an entanglement distribution scheme named Fair-EAS, which prioritizes entanglement allocation and selection for batching multiple requests to maximize overall throughput while maintaining max-min fairness. To facilitate a convenient solution, we transform the nonlinearity of the problem into an equivalent linear programming formulation and decouple the solution into offline and online phases. In the offline phase, we design a multi-round water-filling-like optimization algorithm to determine the optimal path set for predicting entanglement allocation. In the online phase, we introduce an adaptive compensation algorithm and an entanglement “fragment” exhaustion algorithm to dynamically adjust the path set based on successfully generated entangled pairs. Comprehensive simulations show that Fair-EAS outperforms the existing schemes in terms of fairness by significantly enhancing the minimum throughput and throughput deviation among multiple requests while maintaining an overall throughput close to the optimal level.

How to effectively interact audio with vision has garnered considerable interest within the multi-modality research field. Recently, a novel audio-visual video segmentation (AVS) task has been proposed, aiming to segment the sounding objects in video frames under the guidance of audio cues. However, most existing AVS methods are hindered by a modality imbalance where the visual features tend to dominate those of the audio modality, due to a unidirectional and insufficient integration of audio cues. This imbalance skews the feature representation towards the visual aspect, impeding the learning of joint audio-visual representations and potentially causing segmentation inaccuracies. To address this issue, we propose AVSAC. Our approach features a Bidirectional Audio-Visual Decoder (BAVD) with integrated bidirectional bridges, enhancing audio cues and fostering continuous interplay between audio and visual modalities. This bidirectional interaction narrows the modality imbalance, facilitating more effective learning of integrated audio-visual representations. Additionally, we present a strategy for audio-visual frame-wise synchrony as fine-grained guidance of BAVD. This strategy enhances the share of auditory components in visual features, contributing to a more balanced audio-visual representation learning. Extensive experiments show that our method has state-of-the-art performance on several AVS public benchmarks.

Chart images are widely employed as the intuitive form to express information, which renders them highly valuable. Consequently, there is an urgent demand to develop a watermarking algorithm for copyright protection and leakage prevention of chart images. Nevertheless, existing chart watermarking methods fail to thoroughly consider the chart image’s special characteristics and simply rely on the previous natural image-based watermarking framework. Compared to natural images, the chart image generally exhibits relatively simple layouts and textures, containing fewer complex texture regions that watermarks are typically embedded in. Therefore, the embedding locations of watermarks for different distortions can be relatively dispersed in natural images, while for chart images, watermark embedding regions under various distortion conditions tend to be relatively concentrated and share more overlaps. Inspired by the above special characteristics of chart images, to sufficiently leverage them and design a better framework, this paper proposes C3hartMark, a chart watermarking scheme with consecutive-encoding and concurrent-decoding. Instead of using the combined noise layer as existing methods to ensure multiple robustness, a novel consecutive training framework is introduced in this paper, which efficiently utilizes the overlapping of embedded watermark features in chart images, and simultaneously, mitigates the poor convergence brought by the combined noise layer. During the extraction stage, multiple concurrent decoders are introduced to extract the potential embedded watermarks for different distortions independently. Moreover, we also incorporate two special noise layers, namely Captioning and Fusion, to address the corresponding realistic distortions in chart images, and an agnostic noise layer to accommodate potential channel transmission distortions unknown during training. Through extensive experiments, we demonstrate that with the better visual quality, C3hartMark simultaneously outperforms existing state-of-the-art (SOTA) watermarking methods in terms of robustness, achieving 99.57% extraction accuracy under JPEG compression (QF=60).

This paper proposes a novel method of vacatingroom-after-encryption reversible data hiding in encrypted image (VRAE RDHEI), which uses the ideas of channel modeling and channel coding to achieve the enhancement of capacity. The framework of the proposed method maps the processes of data embedding and image recovery to a virtual noisy channel for transmitting special flag bits of image content, and then it uses the systematic polar code to ensure error-free transmission for reversible data hiding. On the data hider side, to reversibly vacate room for secret data, the selected bits of the encrypted image are transformed to flag bits and then encoded to fewer parity bits by systematic polar code. On the receiver side, the secret data can be extracted without error and separate from image recovery. To recover the image, the receiver uses pixel prediction to obtain the noisy flag bits and decodes them to the original flag bits by a special channel knowledge-based decoding algorithm with the parity bits. Then, the original image can be recovered by the flag bits. The experimental results prove that the proposed method outperforms the state-of-the-art VRAE methods.

Recent research in provably secure neural linguistic steganography has overlooked a crucial aspect: the sender must detokenize stegotexts to avoid raising suspicion from the eavesdropper. The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures in all neural language steganography implementations based on these models. Current solutions to this issue involve altering the probability distribution of candidate words, rendering them incompatible with provably secure steganography. We propose a novel secure disambiguation method named SyncPool, which effectively addresses the segmentation ambiguity problem. We group all tokens with prefix relationships in the candidate pool before the steganographic embedding algorithm runs to eliminate uncertainty among ambiguous tokens. To enable the receiver to synchronize the sampling process of the sender, a shared cryptographically-secure pseudorandom number generator (CSPRNG) is deployed to select a token from the ambiguity pool. SyncPool does not change the size of the candidate pool or the distribution of tokens and thus is applicable to provably secure language steganography methods. We provide theoretical proofs and experimentally demonstrate the applicability of our solution to various languages and models, showing its potential to significantly improve the reliability and security of neural linguistic steganography systems.

Large language models(LLMs), represented by ChatGPT, have greatly simplified text generation tasks. However, they have also raised concerns about privacy risks such as data leakage and unauthorized information collection. Existing solutions for privacy-preserving inference face practical challenges related to computational time and communication costs. In this article, we proposeInferDPT, the first practical framework for privacy-preservingInference of closed-box LLMs, implementingDifferentialPrivacy inText generation.InferDPTcomprises two key modules: the “perturbation module” utilizes the differentially private mechanism to generate a perturbed prompt, facilitating privacy-preserving inference with closed-box LLMs; the “extraction module”, inspired by knowledge distillation and phenomenon we observed, extracts coherent and consistent text from the perturbed generation result, ensuring successful text generation completion. To achieve a better balance between utility and privacy protection, we introduce RANTEXT, a novel differentially private mechanism integrated into the perturbation module ofInferDPT, which introduces the concept of “RANdom adjacency list” forTEXTperturbation within the prompt. Experimental results across three datasets demonstrate that the text generation quality ofInferDPTis comparable to that of non-private GPT-4, and RANTEXT surpasses existing state-of-the-art mechanisms, namely, SANTEXT+ and CUSTEXT+ in the trade-off between privacy and utility. Even with a privacy parameter$\varepsilon$value of 6.0, RANTEXT achieves an average privacy protection level of exceeding 0.90 against the embedding inversion attacks, which is 0.58× higher than that of SANTEXT+ and 3.35× higher than that of CUSTEXT+.

Documents are ubiquitous vehicles for information transmission. Beyond the visible content meant for reading, there is a growing interest in hiding additional information in documents. Recent studies have focused on the utilization of glyphs, which are stored in vector format within computer systems. Specifically, glyph variants are manually designed to substitute the original ones in documents, thereby representing information. However, such strategies are costly, only effective for specific font types, and fragile to physical distortions. To address these limitations, this paper presents AutoStegaFont+, a two-stage and dual-modality learning framework designed to synthesize glyph vectors capable of conveying hidden information under real-world distortions. In the first stage, we jointly train an encoder and a decoder with a specialized distortion layer to achieve robust information encoding and decoding of glyph images. Then, the second stage employs a differentiable rasterizer to transfer the information from encoded glyph images to corresponding vectors, enabling the automatic generation of encoded vectors. Extensive experiments demonstrate the robust performance of AutoStegaFont+ across a variety of real-world scenarios, including screenshots, print-camera shooting, and screen-camera shooting, while maintaining compatibility with diverse font types. Additionally, we investigate the information-carrying capacity of individual glyphs, exploring their impact on robustness and visual quality.

Deep learning-based techniques for facial anonymization strive to protect identity details while preserving image usability. We present Antiano, the first work to reveal possible security risks associated with deep learning-based facial anonymization algorithms. By evaluating the entire anonymization process, we introduce a framework comprising four attack types: Anonymization Deactivation Attack, ID Destruction Attack, Downstream Task Attack, and Attribute Attack. These attacks aim to undermine anonymization effectiveness, mislead anonymized outputs and downstream tasks, or modify facial attributes in subtle ways. For each scenario, we develop novel strategies that craft perturbations tailored to a user’s face and transferable across different keys and models. Experimental findings reveal that Antiano achieves significant attack effectiveness against various reversible anonymization models, highlighting security vulnerabilities in current deep learning anonymization algorithms. This paper also distinguishes Antiano from traditional adversarial attacks and provides security recommendations for the future development of facial anonymization technologies.

Digital steganography is the practice of embedding secret information in original normal data to enable covert communication. With the rapid advancement of generative models, generative steganography has gained renewed vitality. As a key medium on the Internet, audio has also become a focus of steganographic research. However, existing audio steganography methods rely on traditional audio synthesis models, which often suffer from suboptimal synthesis quality. In contrast, diffusion models perform well in audio synthesis tasks, but there is a lack of targeted secure audio steganography methods based on them. In addition, existing steganography schemes are generally limited to transmitting only the steganographic object, and other key elements need to be negotiated in advance, which limits their practicality. To address these issues, we propose CoAS, a composite audio steganography method based on text and speech synthesis. Firstly, we use a provably secure linguistic steganography method to embed the synchronous side information required for audio steganography, and then replace the gaussian noise in the diffusion models with message-driven sampling during the audio generation process. Both theoretical analysis and experimental results validate the security and practicality of our composite steganography method in the real world. Audio samples are available at https://meterial.github.io/coas.github.io.

Minimum distortion steganography is currently the mainstream method for modification-based steganography. A key issue in this method is how to define steganographic distortion. With the rapid development of deep learning technology, the definition of distortion has evolved from manual design to deep learning design. Concurrently, rapid advancements in image generation have made generated images viable as cover media. However, existing distortion design methods based on machine learning do not fully leverage the advantages of generated cover media, resulting in suboptimal security performance. To address this issue, we propose GIFDL (Generated Image Fluctuation Distortion Learning), a steganographic distortion learning method based on the fluctuations in generated images. Inspired by the idea of natural steganography, we take a series of highly similar fluctuation images as the input to the steganographic distortion generator and introduce a new GAN training strategy to disguise stego images as fluctuation images. Experimental results demonstrate that GIFDL, compared with state-of-the-art GAN-based distortion learning methods, exhibits superior resistance to steganalysis, increasing the detection error rates by an average of 3.30% across three steganalyzers.

Penetration testing is essential for ensuring Web security by identifying and mitigating vulnerabilities in advance, and the rapid progress of large language models (LLMs) shows great potential to revolutionize this process through intelligent, automated agents. In this work, we establish a comprehensive end-to-end penetration testing benchmark using a real-world penetration testing environment to explore the capabilities of LLM-based agents in this domain. Our results reveal that the agents are familiar to procedures of penetration testing tasks, but they still face limitations in generating accurate commands and executing complete processes. Accordingly, we summarize the current challenges, including the difficulty of maintaining the entire message history and the tendency for the agent to become stuck. Based on the above insights, we propose a Penetration testing State Machine (PSM) that utilizes the Finite State Machine (FSM) methodology to address these limitations. Then, we introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs, which utilizes the inherent inference ability of LLM and the constraint framework of state machines. Our evaluation results show that AutoPT outperforms the the ReAct-based baseline and improves the task completion rate from 22% to 41% on the benchmark target. Compared with the baseline and manual work, AutoPT also reduces time and economic costs further. In general, our AutoPT has facilitated the development of automated penetration testing and bring new findings and insights for both academia and industry.

The technique of hiding secret messages within seemingly harmless covertext to evade examination by censors with rigorous security proofs is known as provably secure steganography (PSS). PSS evolves from symmetric key steganography to public-key steganography, functioning without the requirement of a pre-shared key and enabling the extension to multi-party covert communication and identity verification mechanisms. Recently, a public-key steganography method based on elliptic curves was proposed, which uses point compression to eliminate the algebraic structure of curve points. However, this method has strict requirements on the curve parameters and is only available on half of the points. To overcome these limitations, this paper proposes a more general elliptic curve public key steganography method based on admissible encoding. By applying the tensor square function to the known well-distributed encoding, we construct admissible encoding, which can create the pseudo-random public-key encryption function. The theoretical analysis and experimental results show that the proposed provable secure public-key steganography method can be deployed on all types of curves and utilize all points on the curve.

Medical image segmentation has made significant strides with the development of basic models. Specifically, models that combine CNNs with transformers can successfully extract both local and global features. However, these models inherit the transformer's quadratic computational complexity, limiting their efficiency. Inspired by the recent Receptance Weighted Key Value (RWKV) model, which achieves linear complexity for long-distance modeling, we explore its potential for medical image segmentation. While directly applying vision-RWKV yields suboptimal results due to insufficient local feature exploration and disrupted spatial continuity, we propose a novel nested structure, Zigzag RWKV-in-RWKV (Zig-RiR), to address these issues. It consists of Outer and Inner RWKV blocks to adeptly capture both global and local features without disrupting spatial continuity. We treat local patches as "visual sentences" and use the Outer Zig-RWKV to explore global information. Then, we decompose each sentence into sub-patches ("visual words") and use the Inner Zig-RWKV to further explore local information among words, at negligible computational cost. We also introduce a Zigzag-WKV attention mechanism to ensure spatial continuity during token scanning. By aggregating visual word and sentence features, our Zig-RiR can effectively explore both global and local information while preserving spatial continuity. Experiments on four medical image segmentation datasets of both 2D and 3D modalities demonstrate the superior accuracy and efficiency of our method, outperforming the state-of-the-art method 14.4 times in speed and reducing GPU memory usage by 89.5% when testing on ${1024} \times {1024}$ high-resolution medical images. Our code is available at https://github.com/txchen-USTC/Zig-RiR.

Large-scale image retrieval using deep hashing has become increasingly popular due to the exponential growth of image data and the remarkable feature extraction capabilities of deep neural networks (DNNs). However, deep hashing methods are vulnerable to malicious attacks, including adversarial and backdoor attacks. It is worth noting that these attacks typically involve altering the query images, which is not a practical concern in real-world scenarios. In this paper, we point out that even clean query images can be dangerous, inducing malicious target retrieval results, like undesired or illegal images. To the best of our knowledge, we are the first to study datapoisoningattacks againstdeephashing(PADHASH). Specifically, we first train a surrogate model to simulate the behavior of the target deep hashing model. Then, a strict gradient matching strategy is proposed to generate the poisoned images. Extensive experiments on different models, datasets, hash methods, and hash code lengths demonstrate the effectiveness and generality of our attack method.

Quantum Key Distribution (QKD) technology leverages its inherent security advantages to ensure information-theoretic security for data transmission in networks. However, existing QKD networks still face critical challenges, including network congestion that stems from limited key resources and uneven resource allocation methods. Thus, in this paper, we focus on the issue of network congestion caused by bottleneck links and aim to achieve load balancing. Considering the limited key resources, we first introduce the key resource utilization ratio as an indicator of bottleneck links and formulate the resource allocation problem as an Integer Linear Programming (ILP) problem. To deal with the complexity of the ILP problem, especially in large-scale network scenarios, we design a heuristic algorithm that can obtain near-optimal solutions within polynomial time. Finally, we implement the proposed key resource allocation scheme in various real-world network topologies using a full-stack quantum network simulator. Compared to the existing algorithms, extensive results show that our method can reduce key resource consumption by up to 50% on bottleneck links and improve the robustness of QKD networks when facing burst quantum key agreement requests.

As Deepfake content proliferates online, advancing face manipulation forensics has become crucial. To combat this emerging threat, previous methods mainly focus on studying how to distinguish authentic and manipulated face images. Although impressive, image-level classification lacks explainability and is limited to specific application scenarios, spurring recent research on pixel-level prediction for face manipulation forensics. However, existing forgery localization methods suffer from exploring frequency-based forgery traces in the localization network. In this paper, we observe that multi-frequency spectrum information is effective for identifying tampered regions. To this end, a novel Multi-spectral Class Center Network (MSCCNet) is proposed for face manipulation localization. Specifically, we design a Multi-spectral Class Center (MSCC) module to learn more generalizable and multi-frequency features. Based on the features of different frequency bands, the MSCC module collects multi-spectral class centers and computes pixel-to-class relations. Applying multi-spectral class-level representations suppresses the semantic information of the visual concepts which is insensitive to manipulated regions of forgery images. Furthermore, we propose a Multi-level Features Aggregation (MFA) module to employ more low-level forgery artifacts and structural textures. Meanwhile, we conduct a comprehensive localization benchmark based on pixel-level FF++ and Dolos datasets. Experimental results quantitatively and qualitatively demonstrate the effectiveness and superiority of the proposed MSCCNet. We expect this work to inspire more studies on pixel-level face manipulation localization. The codes are available.

The highly realistic avatar in the metaverse may lead to deepfakes of facial identity. Malicious users can more easily obtain the three-dimensional structure of faces, thus using deepfake technology to create counterfeit videos with higher realism. To automatically discern facial videos forged with the advancing generation techniques, deepfake detectors need to achieve stronger generalization abilities. Inspired by transfer learning, neural networks pre-trained on other large-scale face-related tasks would provide fundamental features for deepfake detection. We propose a video-level deepfake detection method based on a temporal transformer with a self-supervised audio–visual contrastive learning approach for pre-training the deepfake detector. The proposed method learns motion representations in the mouth region by encouraging the paired video and audio representations to be close while unpaired ones to be diverse. The deepfake detector adopts the pre-trained weights and partially fine-tunes on deepfake datasets. Extensive experiments show that our self-supervised pre-training method can effectively improve the accuracy and robustness of our deepfake detection model without extra human efforts. Compared with existing deepfake detection methods, our proposed method achieves better generalization ability in cross-dataset evaluations.

The entanglement transport protocol with a connection-oriented mode ensures the reliable distribution of remote entanglement by reserving dedicated resources on the selected path for users in a quantum network. In most existing protocols, entanglement generation and resource allocation operate with the support of global network-synchronized time slot. However, such synchronization in a large-scale quantum network is challenging, and the idealized time slot model is not conducive to continuous and concurrent requests. Meanwhile, different link performance in memory capacity and entanglement generation rate brings out critical issues, such as long distribution delay and low resource utilization, which has not been adequately addressed by the existing protocols relying on a heuristic adoption of TCP-like transport modes. In light of these observations, we propose a dynamic rate matching-based entanglement transport protocol called DRM-ETP, which allocates different memory units on each link along an entanglement distribution path. Moreover, DRM-ETP incorporates periodic forward and backward interactions to implement fine-grained feedback and a dynamic memory allocation based on priority differentiation. These mechanisms mitigate congestion and unfairness arising from resource contention among burst requests on shared links. Extensive simulation results demonstrate that DRM-ETP significantly outperforms the existing protocols in terms of throughput and resource utilization, with less distribution delay and higher fidelity. Moreover, DRM-ETP exhibits rapid and fair convergence when handling burst requests. Our study opens up possibilities for deploying efficient entanglement transport in quantum networks, thereby holding the promise of enhanced compatibility and novel functionality.

Quantum key distribution (QKD) networks can provide information-theoretically secure key distribution between distant end nodes through key relaying. In QKD networks, the key relay protocol is vital since it provides the coordination specifications between nodes for key relaying and thus directly determines the performance, especially as the network scale expands. However, most existing protocols adopt a synchronous contend-and-relay approach, where the contention and consumption of quantum keys occur simultaneously, neglecting the storable nature of quantum keys and presenting significant challenges in reliability and quantum key utilization. To tackle these challenges, in this paper, we propose an asynchronous key relay protocol (AKRP). AKRP considers the storable nature of quantum keys, and adopts a reserve-then-relay approach to achieve lossless and zero-queuing key relaying through precise management of quantum keys and requests. On this basis, to further improve the performance of the proposed AKRP, we design two enhanced mechanisms, i.e., collision detection and resolution mechanism and multipath routing extension. The former enhances the consensus efficiency of AKRP and provides fine-grained key utilization on each link, and the latter utilizes quantum keys on possible relay paths and thus effectively copes with quantum key exhaustion. By conducting extensive experiments on a semi-physical real QKD network platform, results demonstrate that AKRP is superior to existing schemes in terms of end-to-end key throughput, quantum key consumption, and relaying latency.

Quantum networks, which work by establishing entanglement between distant quantum end nodes (known as end-to-end entanglement distribution), are the promising infrastructure for quantum applications. However, the inherent loss in quantum channels and quantum decoherence contribute to the scarcity of entanglement resources in quantum networks. Consequently, there is an inevitable gap between available entanglement resources and requests’ demands, significantly hindering concurrent end-to-end entanglement distributions. In this paper, we present NarrowGap, an end-to-end entanglement distribution design that can alleviate the negative impact of entanglement resource scarcity on the request service capability of quantum networks. At the heart of NarrowGap, the resource transfer scheme (RTS) is designed to transfer idle entanglement resources to boost the bottlenecks’ capacities based on the unique feature of entanglement swapping, thus narrowing the gap between available entanglement resources and requests’ demands for end-to-end entanglements. Besides, NarrowGap presents a resource allocation scheme (RAS) to guarantee fairness, considering both the success probability of end-to-end entanglement distribution and each request’s demand, to address resource competition in bottlenecks. Extensive simulations demonstrate that NarrowGap outperforms three representative schemes and can achieve more than twice the performance improvement in request service rate.

Infrared small target detection (ISTD) is critical to national security and has been extensively applied in military areas. ISTD aims to segment small target pixels from background. Most ISTD networks focus on designing feature extraction blocks or feature fusion modules, but rarely describe the ISTD process from the feature map evolution perspective. In the ISTD process, the network attention gradually shifts towards target areas. We abstract this process as the directional movement of feature map pixels to target areas through convolution, pooling and interactions with surrounding pixels, which can be analogous to the movement of thermal particles constrained by surrounding variables and particles. In light of this analogy, we propose Thermal Conduction-Inspired Transformer (TCI-Former) based on the theoretical principles of thermal conduction. According to thermal conduction differential equation in heat dynamics, we derive the pixel movement differential equation (PMDE) in the image domain and further develop two modules: Thermal Conduction-Inspired Attention (TCIA) and Thermal Conduction Boundary Module (TCBM). TCIA incorporates finite difference method with PMDE to reach a numerical approximation so that target body features can be extracted. To further remove errors in boundary areas, TCBM is designed and supervised by boundary masks to refine target body features with fine boundary details. Experiments on IRSTD-1k and NUAA-SIRST demonstrate the superiority of our method.

In recent years, with the popularity of social media applications, massive digital images are available online, which brings great convenience to image recreation. However, the use of unauthorized image materials in multi-source composite images is still inadequately regulated, which may cause significant loss and discouragement to the copyright owners of the source image materials. Ideally, deep watermarking techniques could provide a solution for protecting these copyrights based on their encoder-noise-decoder training strategy. Yet existing image watermarking schemes, which are mostly designed for single images, cannot well address the copyright protection requirements in this scenario, since the multi-source image composing process commonly includes distortions that are not well investigated in previous methods, e.g., the extreme downsizing. To meet such demands, we propose MuST, a multi-source tracing robust watermarking scheme, whose architecture includes a multi-source image detector and minimum external rectangle operation for multiple watermark resynchronization and extraction. Furthermore, we constructed an image material dataset covering common image categories and designed the simulation model of the multi-source image composing process as the noise layer. Experiments demonstrate the excellent performance of MuST in tracing sources of image materials from the composite images compared with SOTA watermarking methods, which could maintain the extraction accuracy above 98% to trace the sources of at least 3 different image materials while keeping the average PSNR of watermarked image materials higher than 42.51 dB. We released our code on https://github.com/MrCrims/MuST

The popularity of Machine Learning as a Service (MLaaS) has led to increased concerns about Model Stealing Attacks (MSA), which aim to craft a clone model by querying MLaaS. Currently, most research on MSA assumes that MLaaS can provide soft labels and that the attacker has a proxy dataset with a similar distribution. However, this fails to encapsulate the more practical scenario where only hard labels are returned by MLaaS and the data distribution remains elusive. Furthermore, most existing work focuses solely on stealing the model accuracy, neglecting the model robustness, while robustness is essential in security-sensitive scenarios, e.g, face-scan payment. Notably, improving model robustness often necessitates the use of expensive techniques such as adversarial training, thereby further making stealing robustness a more lucrative prospect. In response to these identified gaps, we introduce a novel Data-Free Hard-Label Robustness Stealing (DFHL-RS) attack in this paper, which enables the stealing of both model accuracy and robustness by simply querying hard labels of the target model without the help of any natural data. Comprehensive experiments demonstrate the effectiveness of our method. The clone model achieves a clean accuracy of 77.86% and a robust accuracy of 39.51% against AutoAttack, which are only 4.71% and 8.40% lower than the target model on the CIFAR-10 dataset, significantly exceeding the baselines. Our code is available at: https://github.com/LetheSec/DFHL-RS-Attack.

Generating realistic human motion from given action descriptions has experienced significant advancements because of the emerging requirement of digital humans. While recent works have achieved impressive results in generating motion directly from textual action descriptions, they often support only a single modality of the control signal, which limits their application in the real digital human industry. This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals, e.g., text and single-frame poses, for generating consecutive human motions by treating multimodal signals as special input tokens in large language models (LLMs). Specifically, we first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction to ask the LLMs to generate the motion answer. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters. To the best of our knowledge, MotionGPT is the first method to generate human motion by multimodal control signals, which we hope can shed light on this new direction. Visit our webpage at https://qiqiapink.github.io/MotionGPT/.

With the flourishing of the Internet, sharing one's photos or automated processing of faces using computer vision technology has become an everyday occurrence. While enjoying the convenience, the concern for identity privacy is also emerging. Therefore, some efforts introduced the concept of ``password'' from traditional cryptography such as RSA into the face anonymization and deanonymization task to protect the facial identity without compromising the usability of the face image. However, these methods either suffer from the poor visual quality of the synthesis results or do not possess the full cryptographic properties, resulting in compromised security. In this paper, we present the first facial identity cryptography framework with full properties analogous to RSA. Our framework leverages the powerful generative capabilities of StyleGAN to achieve megapixel-level facial identity anonymization and deanonymization. Thanks to the great semantic decoupling of StyleGAN's latent space, the identity encryption and decryption process are performed in latent space by a well-designed password mapper in the manner of editing latent code. Meanwhile, the password-related information is imperceptibly hidden in the edited latent code owing to the redundant nature of the latent space. To make our cryptographic framework possesses all the properties analogous to RSA, we propose three types of loss functions: single anonymization loss, sequential anonymization loss, and associated anonymization loss. Extensive experiments and ablation analyses demonstrate the superiority of our method in terms of the quality of synthesis results, identity-irrelevant attributes preservation, deanonymization accuracy, and completeness of properties analogous to RSA.

Text-to-Image person re-identification (TI-ReID) aims to retrieve the images of target identity according to the given textual description. The existing methods in TI-ReID focus on aligning the visual and textual modalities through contrastive feature alignment or reconstructive masked language modeling (MLM). However, these methods parameterize the image/text instances as deterministic embeddings and do not explicitly consider the inherent uncertainty in pedestrian images and their textual descriptions, leading to limited image-text relationship expression and semantic alignment. To address the above problem, in this paper, we propose a novel method that unifies multi-modal uncertainty modeling and semantic alignment for TI-ReID. Specifically, we model the image and textual feature vectors of pedestrian as Gaussian distributions, where the multi-granularity uncertainty of the distribution is estimated by incorporating batch-level and identity-level feature variances for each modality. The multi-modal uncertainty modeling acts as a feature augmentation and provides richer image-text semantic relationship. Then we present a bi-directional cross-modal circle loss to more effectively align the probabilistic features between image and text in a self-paced manner. To further promote more comprehensive image-text semantic alignment, we design a task that complements the masked language modeling, focusing on the cross-modality semantic recovery of global masked token after cross-modal interaction. Extensive experiments conducted on three TI-ReID datasets highlight the effectiveness and superiority of our method over state-of-the-arts.

Large language models (LLMs) have exhibited remarkable capa- bilities in natural language generation, but they have also been observed to magnify societal biases, particularly those related to gender. In response to this issue, several benchmarks have been proposed to assess gender bias in LLMs. However, these bench- marks often lack practical flexibility or inadvertently introduce biases. To address these shortcomings, we introduce GenderCARE, a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics for quantifying and mitigating gender bias in LLMs. To begin, we estab- lish pioneering criteria for gender equality benchmarks, spanning dimensions such as inclusivity, diversity, explainability, objectivity, robustness, and realisticity. Guided by these criteria, we construct GenderPair, a novel pair-based benchmark designed to assess gen- der bias in LLMs comprehensively. Our benchmark provides stan- dardized and realistic evaluations, including previously overlooked gender groups such as transgender and non-binary individuals. Fur- thermore, we develop effective debiasing techniques that incorpo- rate counterfactual data augmentation and specialized fine-tuning strategies to reduce gender bias in LLMs without compromising their overall performance. Extensive experiments demonstrate a significant reduction in various gender bias benchmarks, with re- ductions peaking at over 90% and averaging above 35% across 17 different LLMs. Importantly, these reductions come with minimal variability in mainstream language tasks, remaining below 2%. By offering a realistic assessment and tailored reduction of gender biases, we hope that our GenderCARE can represent a significant step towards achieving fairness and equity in LLMs. More details are available at https://github.com/kstanghere/GenderCARE-ccs24.

Hallucination, posed as a pervasive challenge of multi-modal large language models (MLLMs), has significantly impeded their real-world usage that demands precise judgment. Existing methods mitigate this issue with either training with specific designed data or inferencing with external knowledge from other sources, incurring inevitable additional costs. In this paper, we present OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a Retrospection-Allocation strategy, serving as a nearly free lunch to alleviate the hallucination issue without additional data, knowledge, or training. Our approach begins with an interesting observation that, most hallucinations are closely tied to the knowledge aggregation patterns manifested in the self-attention matrix, i.e., MLLMs tend to generate new tokens by focusing on a few summary tokens, but not all the previous tokens. Such partial overtrust inclination results in the neglecting of image tokens and describes the image content with hallucination. Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue, along with a rollback strategy that retrospects the presence of summary tokens in the previously generated tokens, and re-allocate the token selection if necessary. With extensive experiments, OPERA shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality. Our code is at: https://github.com/shikiw/OPERA.

The rapid advancement of large language models (LLMs) has accelerated the emergence of in-context learning (ICL) as a cutting-edge approach in the natural language processing domain. Recently, ICL has been employed in visual understanding tasks, such as semantic segmentation and image captioning, yielding promising results. However, existing visual ICL framework can not enable producing content across multiple modalities, whicd limits their potential usage scenarios. To address this issue, we present a new ICLframeworkfor visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space, structured as interleaved in-context sequences. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them, facilitating in-context learning. Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline. Experimental re-sults demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines. Overall, our research takes a further step toward unified multimodal in-context learning.

Ethical concerns surrounding copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. One effective solution involves watermarking the generated images. However, existing methods often compromise the model performance or require additional training, which is undesirable for operators and users. To address this issue, we propose Gaussian Shading, a diffusion model watermarking technique that is both performance-lossless and training-free, while serving the dual purpose of copyright protection and tracing of offending content. Our watermark embedding is free of model parameter modifications and thus is plug-and-play. We map the watermark to latent representations following a standard Gaussian distribution, which is indistinguishable from latent representations obtained from the non-watermarked diffusion model. Therefore we can achieve watermark embedding with lossless performance, for which we also provide theoretical proof Furthermore, since the watermark is intricately linked with image semantics, it exhibits resilience to lossy processing and erasure attempts. The watermark can be extracted by Denoising diffusion Implicit Models (DDIM) inversion and inverse sampling. We evaluate Gaussian Shading on multiple versions of Stable Diffusion, and the results demonstrate that Gaussian Shading not only is performance-lossless but also out-performs existing methods in terms of robustness.

High-quality data is crucial for the pre-training performance of large language models.Unfortunately, existing quality filtering methods rely on a known high-quality dataset as reference, which can introduce potential bias and compromise diversity.In this paper, we propose ScalingFilter, a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data, thereby eliminating the influence of the reference dataset in the filtering process.An theoretical analysis shows that ScalingFilter is equivalent to an inverse utilization of scaling laws.Through training models with 1.3B parameters on the same data source processed by various quality filters, we find ScalingFilter can improve zero-shot performance of pre-trained models in downstream tasks.To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations.Extensive experiments reveal that semantic diversity is a reliable indicator of dataset diversity, and ScalingFilter achieves an optimal balance between downstream performance and semantic diversity.1

Large language models (LLMs) have revolutionized the domain of natural language processing because of their excellent performance on various tasks.Despite their impressive capabilities, LLMs also have the potential to generate texts that pose risks of misuse.Consequently, detecting LLM-generated text has become increasingly important.Previous LLMgenerated text detection methods use semantic features, which are stored in the last layer.This leads to methods that overfit the training set domain and exhibit shortcomings in generalization.Therefore, We argue that utilizing intrinsic features rather than semantic features for detection results in better performance.In this work, we design Text Fluoroscopy, a black-box method with better generalizability for detecting LLM-generated text by mining the intrinsic features of the text to be detected.Our method captures the text's intrinsic features by identifying the layer with the largest distribution difference from the last and first layers when projected to the vocabulary space.Our method achieves 7.36% and 2.84% average improvement in detection performance compared to the baselines in detecting texts from different domains generated by GPT-4 and Claude3, respectively.The codes are publicly available at https://github.com/ Fish-and-Sheep/Text-Fluoroscopy.

AI-synthesized text and images have gained significant attention, particularly due to the widespread dissemination of multi-modal manipulations on the internet, which has resulted in numerous negative impacts on society. Existing methods for multi-modal manipulation detection and grounding primarily focus on fusing vision-language features to make predictions, while overlooking the importance of modality-specific features, leading to sub-optimal results. In this paper, we construct a simple and novel transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. To achieve this, we introduce visual/language pre-trained encoders and dual-branch cross-attention (DCA) to extract and fuse modality-unique features. Furthermore, we design decoupled fine-grained classifiers (DFC) to enhance modality-specific feature mining and mitigate modality competition. Moreover, we propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality using learnable queries, thereby improving the discovery of forged details. Extensive experiments on the DGM4dataset demonstrate the superior performance of our proposed model compared to state-of-the-art approaches.

Secret image sharing (SIS) schemes have undergone significant development. However, to the best of our knowledge, none of the existing schemes has considered the deniable property during secret sharing. This presents a problem when we need to share secret images through an untrusted and supervised channel, where we may be coerced to reveal the secret to adversary. Here we propose a deniable SIS (DSIS) scheme. Before sharing the secret image, we manipulate the secret area of the image to create a forged image that possesses deniability. Then we employ SIS to distribute the forged image, while generating an auxiliary matrix derived from secret key. This matrix governs rules for sharing secret area. In the event of coercion to reveal the secret, we have the capability to present the adversary with the forged image instead, thereby retaining control over the disclosure of the secret area at our discretion. In DSIS, we can obtain a secret image with the small-sized secret key losslessly, while we can recover another visually-meaningful fake image to safeguard the secrecy and protect ourselves when facing coercion.

Secret image sharing (SIS) has made significant progress in research and has found wide applications. However, we note that shadows of traditional SIS contain a large amount of redundancy. A novel Shadow-Embeddable Secret Image Sharing scheme (SE-SIS) leveraging the redundancy in the shadows is proposed in this paper. SE-SIS utilizes the random values in Lagrange polynomials of traditional secret image sharing (SIS) scheme, and modifies a shadow to embed another secret image with a secret key. Then other shadows are modified simultaneously according to the properties of Lagrange polynomials to ensure the accurate recovery of the previously shared image. It is worth noting that embedding process does not impact the recovery of the shared image, and the embedded shadow is indistinguishable from the others. SE-SIS modifies noise-liked shadows into other noise-liked ones without affecting the recovery process, thereby achieving a high embedding rate. Meanwhile, SE-SIS realizes lossless recovery for both the shared image and the secret image. Experimental results indicate SE-SIS constructs randomized shadows and exhibits excellent performance in terms of Peak Signal to Noise Ratio (PSNR) and embedding rate.

Recently, vulnerable samples have been shown to be crucial for improving adversarial training performance. Our analysis on existing vulnerable samples mining methods indicate that existing methods have two problems: 1) valuable connections among different pairs of natural samples and their adversarial counterparts are ignored; 2) parts of vulnerable samples are unconsidered. To better leverage vulnerable samples, we propose INter PAir ConstrainT (INPACT) and Vulnerable Aware adveRsarial Training (VART) to address these drawbacks respectively. INPACT assesses adversarial risk with more comprehensive regularization on sample relationships, which takes both inter and intra connections of natural/adversarial sample pairs into consideration. Meanwhile VART makes full use of all vulnerable samples, including notable proportion neglected by existing instance re-weighting strategies. Extensive experiments on different datasets and backbones demonstrate the effectiveness of the proposed method.

Face manipulation has ignited the interests of both academia and industry in very recent years. Existing face manipulation methods can be roughly categorized into two types: face attribute editing and face swapping. In this paper, we focus on swapping the identity. But unlike face swapping which only changes the face region, we attempt at a more challenging task: attribute-aware head swapping. Given a source video and a target video, we replace the whole target head with the whole source head while keeping the original target attributes. To address the inherent appearance gap (e.g., hairstyle, face shape), accompanying background incompatibility and lighting difference, our method consists of three key components: 1) a generative rendering-to-real-head model for source head modeling and attribute transfer; 2) a background modeling network to fix the background incompatibility during head swapping; 3) a deep harmonization network to fix remaining issues and makes the final composited result more realistic. We compare our approach to different face manipulation methods and the experimental results demonstrate its superiority for a lot of challenging cases.

Nowadays, Quantum Key Distribution (QKD) has garnered widespread attention due to its ability to provide symmetric secret keys with information-theoretic security in point-to-point communication. Additionally, key relay technology has been introduced to complete end-to-end key distribution between remote parties by consuming keys in the intermediate links. The routing problem of selecting paths for key relay technology becomes crucial as it directly impacts the network's performance. In this paper, we focus on addressing the routing problem for a specific scenario to serve applications with real-time requirements. Real-time is a critical necessity for supporting various applications, such as video chatting and calling. To satisfy real-time requirements and achieve Quality of Service (QoS) provision to guarantee the completion time, we introduce a distributed routing protocol called Distributed Routing Protocol Based on Key Reservation (Q-RoKR). It reserves keys in advance along the selected path, thereby satisfying real-time requirements. We also propose a priority-awareness mechanism to address resource competition and make efficient use of keys in links. Extensive experiments demonstrate that our protocol effectively meets the real-time requirements and significantly improves throughput. Furthermore, our key efficiency approaches the optimal bound when compared to other comparison schemes.

Large-scale Vision Transformers have achieved promising performance on downstream tasks through feature pre-training. However, the performance of vanilla lightweight Vision Transformers (ViTs) is still far from satisfactory compared to that of recent lightweight CNNs or hybrid networks. In this paper, we aim to unlock the potential of vanilla lightweight ViTs by exploring the adaptation of the widely-used re-parameterization technology to ViTs for improving learning ability during training without increasing the inference cost. The main challenge comes from the fact that CNNs perfectly complement with re-parameterization over convolution and batch normalization, while vanilla Transformer architectures are mainly comprised of linear and layer normalization layers. We propose to incorporate the nonlinear ensemble into linear layers by expanding the depth of the linear layers with batch normalization and fusing multiple linear features with hierarchical representation ability through a pyramid structure. We also discover and solve a new transformer-specific distribution rectification problem caused by multi-branch re-parameterization. Finally, we propose our Two-Dimensional Re-parameterized Linear module (TDRL) for ViTs. Under the popular self-supervised pre-training and supervised fine-tuning strategy, our TDRL can be used in these two stages to enhance both generic and task-specific representation. Experiments demonstrate that our proposed method not only boosts the performance of vanilla Vit-Tiny on various vision tasks to new state-of-the-art (SOTA) but also shows promising generality ability on other networks. Code will be available.

Diffusion models have achieved remarkable success in generating high-quality images. Recently, the open-source models represented by Stable Diffusion (SD) are thriving and are accessible for customization, giving rise to a vibrant community of creators and enthusiasts. However, the widespread availability of customized SD models has led to copyright concerns, like unauthorized model distribution and unconsented commercial use. To address it, recent works aim to let SD models output watermarked content for post-hoc forensics. Unfortunately, none of them can achieve the challenging white-box protection, wherein the malicious user can easily remove or replace the watermarking module to fail the subsequent verification. For this, we propose AquaLoRA as the first implementation under this scenario. Briefly, we merge watermark information into the U-Net of Stable Diffusion Models via a watermark LowRank Adaptation (LoRA) module in a two-stage manner. For watermark LoRA module, we devise a scaling matrix to achieve flexible message updates without retraining. To guarantee fidelity, we design Prior Preserving Fine-Tuning (PPFT) to ensure watermark learning with minimal impacts on model distribution, validated by proofs. Finally, we conduct extensive experiments and ablation studies to verify our design. Our code is available at github.com/Georgefwt/AquaLoRA.

With the rise of social media and the proliferation of facial recognition surveillance, concerns surrounding privacy have escalated significantly. While numerous studies have concentrated on safeguarding users against unauthorized face recognition, a new and often overlooked issue has emerged due to advances in facial restoration techniques: traditional methods of facial obfuscation may no longer provide a secure shield, as they can potentially expose anonymous information to human perception. Our empirical study shows that blind face restoration (BFR) models can restore obfuscated faces with high probability by simply retraining them on obfuscated (e.g., pixelated) faces. To address it, we propose a transferable adversarial obfuscation method for privacy protection against BFR models. Specifically, we observed a common characteristic among BFR models, namely, their capability to approximate an inverse mapping of a transformation from a high-quality image domain to a low-quality image domain. Leveraging this shared model attribute, we have developed a domain-consistent adversarial method for generating obfuscated images. In essence, our method is designed to minimize overfitting to surrogate models during the perturbation generation process, thereby enhancing the generalization of adversarial obfuscated facial images. Extensive experiments on various BFR models demonstrate the effectiveness and transferability of the proposed method.

Binary code analysis plays a pivotal role in various software security applications, such as software maintenance, malware detection, software vulnerability discovery, patch analysis, etc. However, unlike source code, understanding binary code is challenging for reverse engineers due to the absence of semantic information. Therefore, automated tools are needed to assist human players in interpreting binary code. In recent years, two groups of technologies have shown promising prospects: (1) Deep learning-based technologies have demonstrated competitive results in tasks related to binary code understanding, furthermore, (2) Large Language Models (LLMs) have been extensively pre-trained at the source-code level for tasks such as code understanding and generation. This makes participants wonder about the ability of LLMs in binary code understanding. In this work, we propose a benchmark to evaluate the effectiveness of LLMs in real-world reverse engineering scenarios. The benchmark covers two key binary code understanding tasks, including function name recovery and binary code summarization. We gain valuable insights into their capabilities and limitations through extensive evaluations of popular LLMs using our benchmark. Our evaluations reveal that existing LLMs can understand binary code to a certain extent, thereby improving the efficiency of binary code analysis. Our results highlight the great potential of the LLMs in advancing the field of binary code understanding.

Backdoor attacks pose a significant security vulnerability for deep neural networks (DNNs), enabling them to operate normally on clean inputs but manipulate predictions when specific trigger patterns occur. In this paper, we consider a practical post-training scenario backdoor defense, where the defender aims to evaluate whether a trained model has been compromised by backdoor attacks. Currently, post-training backdoor detection approaches often operate under the assumption that the defender has knowledge of the attack information, logit output from the model, and knowledge of the model parameters, limiting their implementation in practical scenarios.

The rapid development of image generative models has lowered the threshold for image creation but also raised security concerns related to the propagation of false information, urgently necessitating the development of detection technologies for AI-generated images. Presently, text-to-image generation stands as the predominant approach to image generation, where the rendering of generated images hinges on two primary factors: text prompts and the inherent characteristics of the model. However, the variety of semantic text prompts yields diverse generated images, posing significant challenges to existing detection methodologies that rely solely on learning from image features, particularly in scenarios with limited samples. To tackle these challenges, this paper presents a novel perspective on the AI-generated image detection task, advocating for detection under semantic-decoupling conditions. Building upon this insight, we propose SemGIR, a semantic-guided image regeneration based method for AI-generated image detection. SemGIR first regenerates images through image-to-text followed by a text-to-image generation process, subsequently utilizing these re-generated image pairs to derive discriminative features. This regeneration process effectively decouples semantic features organically, allowing the detection process to concentrate more on the inherent characteristics of the generative model. Such an efficient detection scheme can also be effectively applied to attribution. Experimental findings demonstrate that in realistic scenarios with limited samples, SemGIR achieves an average detection accuracy 15.76% higher than state-of-the-art (SOTA) methods. Furthermore, in attribution experiments on the SDv2.1 model, SemGIR attains an accuracy exceeding 98%, affirming the effectiveness and practical utility of the proposed method.

Recent studies have revealed that Multi-Sensor Fusion (MSF) algorithms used in autonomous driving localization systems are still vulnerable to sensor spoofing attacks. However, the attack strategies explored in current research have largely remained unchanged, typically involving two fixed stages, as demonstrated in FusionRipper[1]. To address this limitation and investigate new strategies as well as higher attack thresholds, this paper presents an innovative sensor spoofing attack method that utilizes deep reinforcement learning, specifically targeting the widely used Error State Kalman Filter-based MSF localization algorithm. This technique allows for persistent spoofing attacks along a trajectory by training an agent to modify GPS sensor data inputs. In a scenario where the vehicle travels at a constant speed in a straight line, we demonstrate the effectiveness of this approach, achieving a higher attack offset upper limit than the current state-of-the-art methods. This new perspective on injecting false sensor data into the fusion algorithm not only establishes a higher attack threshold but also poses a significant threat to the security of autonomous driving systems.

Large language models (LLMs) have the potential to generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Consequently, detecting whether a text is generated by LLMs has become increasingly important. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. However, since we do not have access to the interior of the black-box model, we must resort to surrogate models, which impacts detection quality. In order to achieve high-quality detection of black-box models, we would like to extract deep intrinsic characteristics of the black-box model generated texts. We view the generation process as a coupled process of prompt and intrinsic characteristics of the generative model. Based on this insight, we propose to decouple prompt and intrinsic characteristics (DPIC) for LLM-generated text detection method. Specifically, given a candidate text, DPIC employs an auxiliary LLM to reconstruct the prompt corresponding to the candidate text, then uses the prompt to regenerate text by the auxiliary LLM, which makes the candidate text and the regenerated text align with their prompts, respectively. Then, the similarity between the candidate text and the regenerated text is used as a detection feature, thus eliminating the prompt in the detection process, which allows the detector to focus on the intrinsic characteristics of the generative model. Compared to the baselines, DPIC has achieved an average improvement of 6.76\% and 2.91\% in detecting texts from different domains generated by GPT4 and Claude3, respectively.

Image steganography aims to conceal secret messages within an image without detection and has a long development history. With the rapid development of generative artificial intelligence, generative image steganography is also thriving. However, existing generative steganography methods struggle to balance steganographic capacity, extraction accuracy, and security. This paper proposes a high-capacity generative steganography method based on the first-order DPM-Solver, called StegaFDS. By utilizing our efficiently designed mapping function, which connects secret messages to the noise space of the diffusion probabilistic model (DPM), we achieve a significant increase in hiding capacity and detection resistance, while maintaining distribution-preserving. To improve message extraction accuracy further, we also optimize the existing first-order DPM-Solver inversion. Additionally, based on pre-trained diffusion models, StegaFDS can generate high-quality stego images without training. Experimental results show that StegaFDS performs exceptionally better than other generative steganography methods in the abovementioned metrics and demonstrates strong potential and availability.

Abstract Efforts have been made to safeguard DNNs from intellectual property infringement. Among different techniques, model fingerprinting has gained popularity due to its ability to examine potential infringement without altering the model’s parameters. However, there is a concern regarding the vulnerability of previous model fingerprints to “ambiguity attacks,” where attackers may use fabricated fingerprints to bypass ownership verification, potentially leading to disputes. To address this issue, we propose a dual-verification-based fingerprint authentication system that incorporates the verification of fingerprint genuineness. Briefly, this system involves two authentication processes: conventional fingerprint methods for authenticating model copyrights and the incorporation of copyright information into the fingerprint feature map to confirm ownership of the model fingerprint. Extensive experiments have been conducted to demonstrate the effectiveness of our approach in resisting ambiguity attacks and managing attempts to remove the fingerprint.

Remote entanglement distribution in an efficient and reliable manner, especially in the context of a large-scale quantum network with multiple requests, remains an unsolved challenge. The key difficulties lie in achieving spontaneous and precise control over the entanglement distribution procedure, as multiple nodes need to reach a consensus on how to perform it. From the network aspect, allocating link-layer entangled pairs as resources to achieve high efficiency is also challenging. To address these issues, we propose a decentralized Reliable Entanglement Distribution Protocol (REDP) for large-scale networks. The protocol operates in a Forward-Backward Propagation (FBP) manner, where consensus is reached hop-by-hop and disseminated to all nodes on the path. We further use probabilistic analysis and quasi-static modeling to seek the fairness and efficiency of the network based on the above transmission model. Accordingly, we introduce a Source Window Strategy (SWS) and an Entanglement Allocation Strategy (EAS) to assign sending windows and allocate resources for multiple requests, ensuring a high level of fairness and efficiency from a network perspective. Through systematic simulations involving both classical and quantum communication protocols, we demonstrate that REDP outperforms existing approaches in terms of fairness, throughput, and fidelity performance.

Transformer based methods have achieved great success in image inpainting recently. However, we find that these solutions regard each pixel as a token, thus suffering from an information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration. 2) They quantize 2563RGB values to a small number (such as 512) of quantized color values. The indices of quantized pixels are used as tokens for the inputs and prediction targets of the transformer. To mitigate these issues, we propose a new transformer based framework called “PUT”. Specifically, to avoid input downsampling while maintaining computation efficiency, we design a patch-based auto-encoder P-VQVAE. The encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by input quantization, an Un-quantized Transformer is applied. It directly takes features from the P-VQVAE encoder as input without any quantization and only regards the quantized tokens as prediction targets.Furthermore, to make the inpainting process more controllable, we introduce semantic and structural conditions as extra guidance. Extensive experiments show that our method greatly outperforms existing transformer based methods on image fidelity and achieves much higher diversity and better fidelity than state-of-the-art pluralistic inpainting methods on complex large-scale datasets (e.g., ImageNet). Codes are available athttps://github.com/liuqk3/PUT.

Image matting is a fundamental and challenging problem in computer vision and graphics. Most existing matting methods leverage a user-supplied trimap as an auxiliary input to produce good alpha matte. However, obtaining high-quality trimap itself is arduous. Recently, some hint-free methods have emerged, however, the matting quality is still far behind the trimap-based methods. The main reason is that, some hints for removing semantic ambiguity and improving matting quality are essential. Apparently, there is a trade-off between interaction cost and matting quality. To balance performance and user-friendliness, we propose an improved deep image matting framework which is trimap-free and only needs sparse user click or scribble interaction to minimize the needed auxiliary constraints while still allowing interactivity. Moreover, we introduce uncertainty estimation that predicts which parts need polishing and conduct uncertainty-guided refinement. To trade off runtime against refinement quality, users can also choose different refinement modes. Experimental results show that our method performs better than existing trimap-free methods and comparably to state-of-the-art trimap-based methods with minimal user effort. Finally, we demonstrate the extensibility of our framework to video human matting without any structure modification, by adding optical flow-based sparse hint propagation and temporal consistency regularization imposed on the single frame.

The intellectual property of deep networks can be easily "stolen" by surrogate model attack. There has been significant progress in protecting the model IP in classification tasks. However, little attention has been devoted to the protection of image processing models. By utilizing consistent invisible spatial watermarks, the work (Zhang et al. 2020) first considered model watermarking for deep image processing networks and demonstrated its efficacy in many downstream tasks. Its success depends on the hypothesis that if a consistent watermark exists in all prediction outputs, that watermark will be learned into the attacker's surrogate model. However, when the attacker uses common data augmentation attacks (e.g., rotate, crop, and resize) during surrogate model training, it will fail because the underlying watermark consistency is destroyed. To mitigate this issue, we propose a new watermarking methodology, "structure consistency", based on which a new deep structure-aligned model watermarking algorithm is designed. Specifically, the embedded watermarks are designed to be aligned with physically consistent image structures, such as edges or semantic regions. Experiments demonstrate that our method is more robust than the baseline in resisting data augmentation attacks. Besides that, we test the generalization ability and robustness of our method to a broader range of adaptive attacks.

Steganographic coding is an essential part of adaptive steganography. There are only two practical near-optimal codes in the context of adaptive steganography so far: Syndrome-Trellis Codes (STCs) based on linear convolutional codes and Steganographic Polar Codes (SPCs) based on polar codes. It can be noticed that both STCs and SPCs are based on channel codes. Like the need for the variety of cryptographic algorithms, to make steganography practical and secure, it is important to devise more adaptive steganographic codes to create more choices for users. Moreover, we want to solve the long-lasting problem of whether lossy source codes-based near-optimal adaptive steganographic coding exists. In this paper, we consider using Low-Density Generator-Matrix (LDGM) codes in adaptive steganography where a new algorithm is proposed. First, we describe the framework of our LDGM codes-based steganographic coding algorithm and establish rigorous upper bounds on average embedding efficiency for individual LDGM steganographic codes with a given information bit degree distribution under the constant distortion profile. Then, we give a provably optimal method of distortion incorporation for adaptive steganography and provide the corresponding log-domain Belief Propagation Guided Decimation (log-BPGD) algorithm to minimize the additive distortion. The syndrome coding technique is applied to realize definitive encoding and decoding of the secret message. We report experiments for various distortion profiles, payload rates, and code lengths. The results verify the near-optimal performance of the proposed method, by which the possibility of designing near-optimal adaptive steganographic coding methods based on lossy source coding is confirmed.

The development of generative AI applications has revolutionized the data environment for steganography, providing a new source of steganographic cover. However, existing generative data-based steganography methods typically require white-box access, rendering them unsuitable for black-box generative models. To overcome this limitation, we propose a novel steganography method for generated images, which leverages the volatility of generative models and is applicable in black-box scenarios. The volatility of generative models refers to the ability to generate a series of images with slight variations by fine-tuning the input parameters of the model. These generated images exhibit varying degrees of volatility in different areas. To resist steganalysis, we mask steganographic modifications by confusing them with the inherent volatility of the model. Specifically, by modeling distributions of generated pixels and estimating the parameters of the distributions, the occurrence probabilities of generated pixels can be obtained, which serve as an effective measure for steganographic modification probabilities to render stego images as indistinguishable as possible from the images producible by the model. Moreover, we further combine it with existing costs to develop a more comprehensive steganographic algorithm. Experimental results show that the proposed method significantly outperforms baseline and comparative methods in resisting both feature-based and CNN-based steganalyzers.

In recent years, steganalysis based on deep learning has evolved rapidly. However, training deep learning models is data-consuming. The models are prone to overfitting when data is limited. Data augmentation is an effective method to mitigate overfitting. Existing data augmentation methods in steganalysis can be categorized into cover enrichment and virtual augmentation. They are used in different stages. Cover enrichment refers to introducing additional cover-stego pairs in some ways, which is performed prior to training. In contrast, virtual augmentation augments data during training. Existing virtual augmentation methods are designed heuristically and rely on expert knowledge. In this paper, we propose the first automatic virtual data augmentation method for steganalysis. Specifically, we design an augmentation network that augments cover and stego images by intelligently adding noises. The augmentation network is trained adversarially with the steganalyzer to generate diverse data. Meanwhile, a “class-invariant” module prevents the augmentation network from changing the original data distribution too much. A “stabilizer” loss function is designed that keeps the adversarial training stable by constraining the number of noises. The experimental results show that the proposed method outperforms existing virtual augmentation methods. Moreover, combining the proposed method and cover enrichment can further boost performance.

Infrared small target detection (ISTD) aims to segment small target pixels from infrared images and has extensive applications in many fields. Despite multiple progress, challenges remain as present methods still easily suffer from missed detection. Also, present methods are not sensitive enough to irregular target shapes. We argue that the main reason is that some informative small target features get lost during the aggressive downsampling in the encoder without effective recovery. In this article, we propose a new network with a dual-branch encoder-decoder structure for ISTD to address the two challenges. Specifically, to better preserve small target body features for more accurate target locations, we propose to maintain a relatively high resolution of feature maps in one encoder branch. For the other encoder branch, we gradually enlarge feature channels while shrinking resolutions and devise Perona-Malik diffusion (PMD) blocks to preserve shape cues inspired by the shape-preserving effect of PMD in denoising. The encoded high-resolution target body features and high-channel shape cues actually complement each other, so we design channel-resolution interact modules (CRIMs) to combine them. In the decoder, we propose orthogonal central difference fusion (OCDF) that relies on mining contrast differences to further refine shape-aware ISTD quality. Experiments on NUAA-SIRST and IRSTD-1k prove the superiority of our method.

Recently, infrared small-target detection (ISTD) has made significant progress, thanks to the development of basic models. Specifically, the models combining CNNs with Transformers can successfully extract both local and global features. However, the disadvantage of the Transformer is also inherited, that is, the quadratic computational complexity to sequence length. Inspired by the recent basic model with linear complexity for long-distance modeling, Mamba, we explore the potential of this state-space model (SSM) for ISTD tasks in terms of effectiveness and efficiency in the article. However, directly applying Mamba achieves suboptimal performances due to the insufficient harnessing of local features, which are imperative for detecting small targets. Instead, we tailor a nested structure, Mamba-in-Mamba (MiM-ISTD), for efficient ISTD. It consists of Outer and Inner Mamba blocks to adeptly capture both global and local features. Specifically, we treat the local patches as “visual sentences” and use the Outer Mamba to explore the global information. We then decompose each visual sentence into subpatches as “visual words” and use the Inner Mamba to further explore the local information among words in the visual sentence with negligible computational costs. By aggregating the visual word and visual sentence features, our MiM-ISTD can effectively explore both global and local information. Experiments on NUAA-SIRST and IRSTD-1k show the superior accuracy and efficiency of our method. Specifically, MiM-ISTD is$8\times $faster than the SOTA method and reduces GPU memory usage by 62.2% when testing on$2048 \times 2048$images, overcoming the computation and memory constraints on high-resolution infrared images.

Image steganography is a technique to conceal secret messages within digital images. Steganalysis, on the contrary, aims to detect the presence of secret messages within images. Recently, deep-learning-based steganalysis methods have achieved excellent detection performance. As a countermeasure, adversarial steganography has garnered considerable attention due to its ability to effectively deceive deep-learning-based steganalysis. However, steganalysts often employ unknown steganalytic models for detection. Therefore, the ability of adversarial steganography to deceive non-target steganalytic models, known as transferability, becomes especially important. Nevertheless, existing adversarial steganographic methods do not consider how to enhance transferability. To address this issue, we propose a novel adversarial steganographic scheme named Natias. Specifically, we first attribute the output of a steganalytic model to each neuron in the target middle layer to identify critical features. Next, we corrupt these critical features that may be adopted by diverse steganalytic models. Consequently, it can promote the transferability of adversarial steganography. Our proposed method can be seamlessly integrated with existing adversarial steganography frameworks. Thorough experimental analyses affirm that our proposed technique possesses improved transferability when contrasted with former approaches, and it attains heightened security in retraining scenarios.

The progression of text-to-image generation models has incited an upsurge in disseminating generated images across social networks, providing a fertile ground for steganography. Presently, the majority of generated images are crafted utilizing black-box APIs and social networks employ lossy compression on uploaded images. However, there is a dearth of steganographic research conducted on black-box generated images, and the distinctive attributes of the generation model have not been harnessed, resulting in a performance that fails to achieve both security and robustness simultaneously. To address these challenges, we propose an innovative steganographic framework, Steganography based on Concomitantly shaRing generated Images and PrompTs (SCRIPT). This framework ensures security and robustness by precisely identifying robust coefficients within the image for message embedding and synchronizing their positions. For precise identification, we assess the ability of coefficients to withstand unknown spatial perturbations, employing this metric to quantify their robustness. For positional synchronization of robust coefficients, the relevant prompts are uploaded alongside the stego image, allowing the recipient to reconstruct the cover image using a mutually agreed random seed and the provided prompt. Subsequently, positional synchronization is achieved by consistently adopting an identical method for selecting robust coefficients between the sender and the recipient. By amalgamating these strategies, SCRIPT significantly surpasses prior algorithms. Empirical results validate our approach, with a noteworthy 98% message extraction success rate and a substantial 20%+ enhancement in security across diverse payloads.

Steganography is the technique of hiding secret messages within seemingly harmless covers to elude examination by censors. Despite having been proposed several decades ago, provably secure steganography has not gained popularity among researchers due to its rigorous data requirements. Recent advancements in generative models have enabled these researchers to provide explicit data distributions, which has contributed to the development of provably secure steganography methods. However, these methods depend on the assumption of a preshared key. In practical settings, these methods face various challenges, including key agreement, key updating, and user expansion. Although public-key steganography provides a viable solution, existing public-key steganography approaches are burdened with inefficiency and complex implementation in practical scenarios. In this paper, we proposes a practical public-key steganography method based on elliptic curve cryptography and a generative model. This method is the first comprehensive and practical approach to public-key steganography and steganographic key exchange. Additionally, we provide a specific instance to illustrate the proposed method. The security of the proposed construction is also proven based on computational complexity theory. Further experiments have demonstrated the security and efficiency of the proposed method.

The rapid development of large language models (LLMs) has yielded impressive success in various downstream tasks. However, the vast potential and remarkable capabilities of LLMs also raise new security and privacy concerns if they are exploited for nefarious purposes due to their open-endedness. For example, LLMs may be used to plagiarize or imitate writing, thereby infringing the copyright of the original content or to create indiscriminate fake information based on a certain source text. In some cases, LLMs can even analyze text from the Internet to infer personal privacy. Unfortunately, previous text protection research could not foresee the emergence of powerful LLMs, rendering it no longer effective in this new context. To bridge this gap, we introduce Silent Guardian (SG), a text protection mechanism against LLMs, which allows LLMs to refuse to generate responses when receiving protected text, preventing the malicious use of text from the source. Specifically, we first propose the concept of Truncation Protection Examples (TPE). By carefully modifying the text to be protected, TPE can induce LLMs to first sample the end token, thus directly terminating the interaction. In addition, to efficiently construct TPE in the discrete space of text data, we propose a novel optimization algorithm called Super Tailored Protection (STP), which is not only highly efficient but also maintains the semantic consistency of the text during the optimization process. The comprehensive experimental evaluation demonstrates that SG can effectively protect the target text under various configurations and achieve almost 100% protection success rate in some cases. Notably, SG also exhibits relatively good transferability and robustness, making its application in practical scenarios possible. Our code is available athttps://github.com/weiyezhimeng/Silent-Guardian.

Notwithstanding the prominent performance shown in various applications, point cloud recognition models have often suffered from natural corruptions and adversarial perturbations. In this paper, we delve into boosting the general robustness of point cloud recognition, proposing Point-Cloud Contrastive Adversarial Training (PointCAT). The main intuition of PointCAT is encouraging the target recognition model to narrow the decision gap between clean point clouds and corrupted point clouds by devising feature-level constraints rather than logit-level constraints. Specifically, we leverage a supervised contrastive loss to facilitate the alignment and the uniformity of hypersphere representations, and design a pair of centralizing losses with dynamic prototype guidance to prevent features from deviating outside their belonging category clusters. To generate more challenging corrupted point clouds, we adversarially train a noise generator concurrently with the recognition model from the scratch. This differs from previous adversarial training methods that utilized gradient-based attacks as the inner loop. Comprehensive experiments show that the proposed PointCAT outperforms the baseline methods, significantly enhancing the robustness of diverse point cloud recognition models under various corruptions, including isotropic point noises, the LiDAR simulated noises, random point dropping, and adversarial perturbations. Our code is available at: https://github.com/shikiw/PointCAT.

Image restoration under adverse weather conditions (e.g., rain, snow, and haze) is a fundamental computer vision problem that has important implications for various downstream applications. Distinct from early methods that are specially designed for specific types of weather, recent works tend to simultaneously remove various adverse weather effects based on either spatial feature representation learning or semantic information embedding. Inspired by various successful applications incorporating large-scale pre-trained models (e.g., CLIP), in this paper, we explore their potential benefits for leveraging large-scale pre-trained models in this task based on both spatial feature representation learning and semantic information embedding aspects: 1) spatial feature representation learning, we design a Spatially Adaptive Residual (SAR) encoder to adaptively extract degraded areas. To facilitate training of this model, we propose a Soft Residual Distillation (CLIP-SRD) strategy to transfer spatial knowledge from CLIP between clean and adverse weather images; 2) semantic information embedding, we propose a CLIP Weather Prior (CWP) embedding module to enable the network to adaptively respond to different weather conditions. This module integrates the sample-specific weather priors extracted by the CLIP image encoder with the distribution-specific information (as learned by a set of parameters) and embeds these elements using a cross-attention mechanism. Extensive experiments demonstrate that our proposed method can achieve state-of-the-art performance under various and severe adverse weather conditions. The code will be made available.

Database watermarking can be used for copyright verification and leakage traceability, effectively protecting the security of the database. However, the existing watermarking schemes commonly embed watermarks by modifying the original data, which changes the statistical characteristics and affects the statistical analysis of the database. Therefore, this paper proposes SCPW, aStatisticalCharacteristicsPreserving robust databaseWatermarking framework. First, we perform a theoretical analysis and propose a data modification scheme maintaining the statistical characteristics unchanged. Then, we establish the correspondence between the data and the watermarks that need to be embedded in it by grouping. Finally, the watermark message is embedded into the database through data verification and modification. Specifically, for data that needs to be watermarked, we first verify whether the potential watermark bits extracted from the data are the same as bits that need to be embedded. If they are the same, we regard this original data, usually a floating point number, as a “good number” and do not modify it. Otherwise, we modify the data until it becomes a “good number” using a data modification scheme that preserves the statistical characteristics proposed by the theoretical analysis. In addition, we also use the genetic algorithm to optimize the grouping results and increase the proportion of “good number”, thereby reducing the proportion of data that needs to be modified and further reducing distortion. To our best knowledge, SCPW is the first watermarking scheme that ensures the preservation of statistical characteristics, and the experimental results also prove its excellent ability to preserve statistical characteristics compared to existing schemes. Moreover, experiments also illustrate that our method is robust against a wide range of attacks. When under deletion attack (deletion rate = 90%), the bit error rate of watermark extraction is only 0.8%, which is more than 12% lower than the current best method.

The maturity of generative models and the popularity of generated data have brought new technical means and camouflage environments to steganography. Numerous generative image steganography methods have emerged, but achieving provable security, robustness, and relatively high capacity simultaneously remains challenging. This paper proposes a provably secure robust image steganography method via the generative adversarial network (GAN), named PARIS. The sender maps the secret message, following a uniform distribution, to latent vectors conforming to a standard Gaussian distribution using inverse transform sampling. Subsequently, the latent vector is fed into the generator, producing the stego image. In this way, the stego image cannot be distinguished from the normally generated image. The receiver extracts the secret message from the recovered latent vector via gradient descent optimization. To enhance the robustness, a noise layer is introduced while recovering the latent vector to simulate potential lossy operations in real scenarios. The security of the proposed method is theoretically proven. Extensive experiments have also verified the proposed method's robustness, security, and relatively high capacity in terms of different GAN architectures, noises, and datasets.

In recent years, considerable progress has been witnessed in the person re-identification (Re-ID). However, in a more realistic long-term scenario, the appearance shift arising from the clothes-changing inevitably deteriorates the conventional methods that heavily depend on the clothing color. Although the current clothes-changing person Re-ID methods introduce external human knowledge (i.e, contour, mask) and sophisticated feature decoupling strategy to alleviate the clothing shift, they still face the risk of overfitting to clothing due to the limited clothing diversity of training set. To more efficiently and effectively promote the clothes-irrelevant feature learning, we present a novel joint Identity-aware Mixstyle and Graph-enhanced Prototype method for clothes-changing person Re-ID. Specifically, by treating the cloth-changing as fine-grained domain/style shift, the identity-aware mixstyle (IMS) is proposed from the perspective of domain generalization, which mixes the instance-level feature statistics of samples within each identity to synthesize novel and diverse clothing styles, while retaining the correspondence between synthesized samples and latent label space. By incorporating the IMS module, the more diverse styles can be exploited to train a clothing-shift robust model. To further reduce the feature discrepancy caused by clothing variations, the graph-enhanced prototype constraint (GEP) module is proposed to explore the graph similarity structure of style-augmented samples across memory bank to build informative and robust prototypes, which serve as powerful exemplars for better clothing-irrelevant metric learning. The two modules are integrated into a joint learning framework and benefit each other. The extensive experiments conducted on clothes-changing person Re-ID datasets validate the superiority and effectiveness of our method. In addition, our method also shows good universality and corruption robustness on other Re-ID tasks.

Entanglement distribution between distant quantum nodes plays an essential role in realizing quantum networks’ capabilities. In addition to path selection, remote entanglement distribution involves two pivotal quantum operations, i.e., entanglement generation and entanglement swapping. The existing studies mainly adopt two methods, i.e., Tell-and-Generation (TAG) and Tell-and-Swapping (TAS), to manage these two quantum operations on a selected path. However, both methods fatally introduce redundant stop-and-wait processes, which are detrimental to the performance of remote entanglement distribution in terms of latency and fidelity. To achieve low-latency and high-fidelity entanglement distribution between far-off quantum nodes, we propose a segment-based method consisting of an entanglement generation algorithm and a segment design to diminish the unnecessary stop-and-wait processes. The entanglement generation algorithm adopts a concurrent design to establish entanglement links using the one-demand generation model, thus effectively reducing waiting time compared to hop-by-hop and parallel designs. The segment design is proposed to split a long-distance path into multiple short-haul segments with the similar ability to swap entanglement, and these segments build multi-hop entanglement connections in parallel. Extensive simulations show that the segment-based method significantly outperforms the existing methods, including TAG and TAS, in entanglement distribution latency and effectively mitigates fidelity attenuation.

The quantum network that allows users to communicate in a quantum way will be available in the foreseeable future. The network capable of distributing Bell state entangled pairs faces many challenges due to entanglement decoherence and limited network performance, especially when the network scale is enormous. Many entanglement distribution protocols have been proposed so far, and most of them are in a centralized and synchronized manner, which may be infeasible in large-scale networks. As such, in this paper, we propose a full spontaneous version of quantum networks in which the quantum nodes autonomously manage multiple entanglement distribution requests. However, one major issue is that quantum nodes have little knowledge about the network, especially the congestion (e.g., some nodes may have no usable quantum memories). We present a routing algorithm to adaptive evaluate the congestion on the neighbor nodes to avoid potential congestion. We use SimQN, the new network layer simulation platform built by our research team, to evaluate our proposed design. The result demonstrates that it can adapt to changes in network resources and reduce the drop rate that eventually leads to a higher entanglement distribution rate but remains fair for multiple requests to use the network resources fairly and achieve a more balanced throughput.

This paper explores a better prediction target for BERT pre-training of vision transformers. We observe that current prediction targets disagree with human perception judgment. This contradiction motivates us to learn a perceptual prediction target. We argue that perceptually similar images should stay close to each other in the prediction target space. We surprisingly find one simple yet effective idea: enforcing perceptual similarity during the dVAE training. Moreover, we adopt a self-supervised transformer model for deep feature extraction and show that it works well for calculating perceptual similarity. We demonstrate that such learned visual tokens indeed exhibit better semantic meanings, and help pre-training achieve superior transfer performance in various downstream tasks. For example, we achieve 84.5% Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming the competitive method BEiT by +1.3% under the same pre-training epochs. Our approach also gets significant improvement on object detection and segmentation on COCO and semantic segmentation on ADE20K. Equipped with a larger backbone ViT-H, we achieve the state-of-the-art ImageNet accuracy (88.3%) among methods using only ImageNet-1K data.

Audio watermarking is widely used for leaking source tracing. The robustness of the watermark determines the traceability of the algorithm. With the development of digital technology, audio re-recording (AR) has become an efficient and covert means to steal secrets. AR process could drastically destroy the watermark signal while preserving the original information. This puts forward a new requirement for audio watermarking at this stage, that is, to be robust to AR distortions. Unfortunately, none of the existing algorithms can effectively resist AR attacks due to the complexity of the AR process. To address this limitation, this paper proposes DeAR, a deep-learning-based audio re-recording resistant watermarking. Inspired by DNN-based image watermarking, we pioneer a deep learning framework for audio carriers, based on which the watermark signal can be effectively embedded and extracted. Meanwhile, in order to resist the AR attack, we delicately analyze the distortions that occurred in the AR process and design the corresponding distortion layer to cooperate with the proposed watermarking framework. Extensive experiments show that the proposed algorithm can resist not only common electronic channel distortions but also AR distortions. Under the premise of high-quality embedding (SNR=25.86dB), in the case of a common re-recording distance (20cm), the algorithm can effectively achieve an average bit recovery accuracy of 98.55%.

Hiding information in text documents has been a hot topic recently, with the most typical schemes of utilizing fonts. By constructing several fonts with similar appearances, information can be effectively represented and embedded in documents. However, due to the unstructured characteristic, font vectors are more difficult to synthesize than font images. Existing methods mainly use handcrafted features to design the fonts manually, which is time-consuming and labor-intensive. Moreover, due to the diversity of fonts, handcrafted features are not generalizable to different fonts. Besides, in practice, since documents might be distorted through transmission, ensuring extractability under distortions is also an important requirement. Therefore, three requirements are imposed on vector font generation in this domain: automaticity, generalizability, and robustness. However, none of the existing methods can satisfy these requirements well and simultaneously. To satisfy the above requirements, we propose AutoStegaFont, an automatic vector font synthesis scheme for hiding information in documents. Specifically, we design a two-stage and dual-modality learning framework. In the first stage, we jointly train an encoder and a decoder to invisibly encode the font images with different information. To ensure robustness, we target designing a noise layer to work with the encoder and decoder during training. In the second stage, we employ a differentiable rasterizer to establish a connection between the image and the vector modality. Then, we design an optimization algorithm to convey the information from the encoded image to the corresponding vector. Thus the encoded font vectors can be automatically generated. Extensive experiments demonstrate the superior performance of our scheme in automatically synthesizing vector fonts for hiding information in documents, with robustness to distortions caused by low-resolution screenshots, printing, and photography. Besides, the proposed framework has better generalizability to fonts with diverse styles and languages.

Model inversion (MI) attacks have raised increasing concerns about privacy, which can reconstruct training data from public models. Indeed, MI attacks can be formalized as an optimization problem that seeks private data in a certain space. Recent MI attacks leverage a generative adversarial network (GAN) as an image prior to narrow the search space, and can successfully reconstruct even the high-dimensional data (e.g., face images). However, these generative MI attacks do not fully exploit the potential capabilities of the target model, still leading to a vague and coupled search space, i.e., different classes of images are coupled in the search space. Besides, the widely used cross-entropy loss in these attacks suffers from gradient vanishing. To address these problems, we propose Pseudo Label-Guided MI (PLG-MI) attack via conditional GAN (cGAN). At first, a top-n selection strategy is proposed to provide pseudo-labels for public data, and use pseudo-labels to guide the training of the cGAN. In this way, the search space is decoupled for different classes of images. Then a max-margin loss is introduced to improve the search process on the subspace of a target class. Extensive experiments demonstrate that our PLG-MI attack significantly improves the attack success rate and visual quality for various datasets and models, notably, 2 ∼ 3× better than state-of-the-art attacks under large distributional shifts. Our code is available at: https://github.com/LetheSec/PLG-MI-Attack.

This paper presents a simple yet effective framework MaskCLIP, which incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. The core idea of masked self-distillation is to distill representation from a full image to the representation predicted from a masked image. Such incorporation enjoys two vital benefits. First, masked self-distillation targets local patch representation learning, which is complementary to vision-language contrastive focusing on text-related representation. Second, masked self-distillation is also consistent with vision-language contrastive from the perspective of training objective as both utilize the visual encoder for feature aligning, and thus is able to learn local semantics getting indirect supervision from the language. We provide specially designed experiments with a comprehensive analysis to validate the two benefits. Symmetrically, we also introduce the local semantic supervision into the text branch, which further improves the pretraining performance. With extensive experiments, we show that MaskCLIP, when applied to various challenging downstream tasks, achieves superior results in linear probing, finetuning, and zeroshot performance with the guidance of the language encoder. Code will be release at https://github.com/LightDXY/MaskCLIP.

We present Diversity-Aware Meta Visual Prompting (DAM-VP), an efficient and effective prompting method for transferring pre-trained models to downstream tasks with frozen backbone. A challenging issue in visual prompting is that image datasets sometimes have a large data diversity whereas a per-dataset generic prompt can hardly handle the complex distribution shift toward the original pretraining data distribution properly. To address this issue, we propose a dataset Diversity-Aware prompting strategy whose initialization is realized by a Meta-prompt. Specifically, we cluster the downstream dataset into small homogeneity subsets in a diversity-adaptive way, with each subset has its own prompt optimized separately. Such a divide-and-conquer design reduces the optimization difficulty greatly and significantly boosts the prompting performance. Furthermore, all the prompts are initialized with a meta-prompt, which is learned across several datasets. It is a bootstrapped paradigm, with the key observation that the prompting knowledge learned from previous datasets could help the prompt to converge faster and perform better on a new dataset. During inference, we dynamically select a proper prompt for each input, based on the feature distance between the input and each subset. Through extensive experiments, our DAM-VP demonstrates superior efficiency and effectiveness, clearly surpassing previous prompting methods in a series of downstream datasets for different pretraining models. Our code is available at: https://github.com/shikiw/DAM-VP.

Canonical binary AIFV coding [1] contains two trees T0and T1. We show the method to compress T0, and the method to compress T1is with a similar way. We provide a new method to store the number of leaves, master nodes and complete internal nodes in each layer and compactly encode the string of numbers according to the specific property between the nodes.

Infrared small target detection (ISTD) is indispensable in remote sensing and military surveillance. Existing ISTD methods can discover regularly-shaped and clear objects well, but tend to overlook the tough-to-detect ones, such as targets with irregular shapes or blurry boundaries, causing inaccurate segmentation and missed detection. Considering that boundary areas assemble rich uncertainty information, we propose the Boundary-Aware Uncertainty Enhanced Network (BAUENet), where Uncertainty Enhanced Context Refinement (UECR) and Adaptive Feature Fusion Modules (AFFM) are devised to address this problem. Specifically, UECR extracts spatial contexts and refines them with uncertain area maps derived from backbone intermediate outputs, so as to distinguish boundary areas from other regions. AFFM adaptively aggregates cross-level features via balancing low-level details and high-level semantics for finer boundary preservation in both channel and spatial dimensions during up-sampling feature fusion. Experiments on several public datasets demonstrate the effectiveness of the proposed method, especially for irregular shape and blurry boundary cases.

Image steganography is the technique of concealing secret messages into digital images without arousing suspicion from detectors. Recently, adversarial steganography has received much attention from the research community, since it is effective in deceiving target deep-learning-based steganalysis (DLS) and designing more secure embedding distortion. However, how to combine adversarial steganography with handcrafted adjustment strategies to design adversarial steganography based on joint distortion has not been discussed yet. In this paper, incorporating adversarial steganography and joint distortion assignment, we present a novel adversarial steganographic scheme named JAS (Joint Adversarial Steganography). We compute joint distortion and adjust it based on joint gradient, which is a vector consisting of the gradients of two adjacent pixels, until the resulting stego image could deceive the target DLS. Furthermore, by combining JAS with synchronizing modification directions profile, we enhance the steganography security more desirably. Experiments demonstrate that the proposed method effectively enhances the anti-detection ability of joint distortion steganography.

Weakly-supervised Temporal Action Localization (WTAL) aims at localizing actions in untrimmed videos with only video-level labels. Most existing methods embrace a "localization by classification" paradigm and adopt a model that pre-trained with recognition task for feature extraction. The gap between recognition and localization tasks leads to inferior performance. Some recent works attempt to utilize feature enhancement to obtain better feature for localization and boost the performance to some extent. However, they are limited to intra-video information exploiting, while ignoring meaningful inter-video information in the dataset. In this paper, we propose a novel Dual-Feature Enhancement (DFE) method for WTAL, which can utilize both intra-and inter-video information. For intra-video, a local feature enhancement module is designed to promote the feature interaction along the temporal dimension within each video. For inter-video information, a global memory module is firstly designed to learn the representations for different categories across different videos. Then, a global feature enhancement module is used to enhance the video features with the help of those global representations in the memory. Besides, to reduce the extra computational cost caused by global enhancement module in the inference stage, a distillation loss is applied to enforce the local branch to learn the information from global branch, so the global enhancement module could be removed during inference. The proposed method achieves state-of-the-art performance on popular benchmarks.

Unsupervised Domain Adaptative person re-identification (UDA ReID) aims to transfer the knowledge of pre-trained model from labeled source domain to unlabeled target domain. Although the current clustering-based methods have achieved promising success, they neglect the tolerance of the model to cope with different-level noise, which may cause the model to memorize some incorrect patterns caused by label noise and overfit on them rapidly in the early stages. In this paper, we introduce a novel Dual Uncertainty guided Curriculum Learning (DUCL) method to tackle the above problems. Specifically, the reliability-based curriculum allocation is proposed to enforce the sample adaptation in an easy-to-hard manner, which is further assisted by a novel dual-uncertainty re-weighting strategy to alleviate the influence of label noise. In addition, we design Part-aware Feature Refinement (PAFR) to enhance the discrimination of model and thereby acquiring more reliable pseudo-labels. Specifically, the part-aware attention maps are exploited in the PAFR to integrate fine-grained semantics into holistic representation. Extensive experiments have validated the superiority of the proposed method.

Nowadays, social media has become the preferred communication platform for web users but brought security threats. Linguistic steganography hides secret data into text and sends it to the intended recipient to realize covert communication. Compared to edit-based linguistic steganography, generation-based approaches largely improve the payload capacity. However, existing methods can only generate stego text alone. Another common behavior in social media is sending semantically related image-text pairs. In this paper, we put forward a novel image captioning-based stegosystem, where the secret messages are embedded into the generated captions. Thus, the semantics of the stego text can be controlled and the secret data can be transmitted by sending semantically related image-text pairs. To balance the conflict between payload capacity and semantic preservation, we proposed a new sampling method called Two-Parameter Semantic Control Sampling to cutoff low-probability words. Experimental results have shown that our method can control diversity, payload capacity, security, and semantic accuracy at the same time.

Transformer is popular in recent 3D human pose estimation, which utilizes long-term modeling to lift 2D keypoints into the 3D space. However, current transformer-based methods do not fully exploit the prior knowledge of the human skeleton provided by the kinematic structure. In this paper, we propose a novel transformer-based model EvoPose to introduce the human body prior knowledge for 3D human pose estimation effectively. Specifically, a Structural Priors Representation (SPR) module represents human priors as structural features carrying rich body patterns, e.g. joint relationships. The structural features are interacted with 2D pose sequences and help the model to achieve more informative spatiotemporal features. Moreover, a Recursive Refinement (RR) module is applied to refine the 3D pose outputs by utilizing estimated results and further injects human priors simultaneously. Extensive experiments demonstrate the effectiveness of EvoPose which achieves a new state of the art on two most popular benchmarks, Human3.6M and MPI-INF-3DHP.

In this paper, we investigate the adversarial robustness of vision transformers that are equipped with BERT pretraining (e.g., BEiT, MAE). A surprising observation is that MAE has significantly worse adversarial robustness than other BERT pretraining methods. This observation drives us to rethink the basic differences between these BERT pretraining methods and how these differences affect the robustness against adversarial perturbations. Our empirical analysis reveals that the adversarial robustness of BERT pretraining is highly related to the reconstruction target, i.e., predicting the raw pixels of masked image patches will degrade more adversarial robustness of the model than predicting the semantic context, since it guides the model to concentrate more on medium-/high-frequency components of images. Based on our analysis, we provide a simple yet effective way to boost the adversarial robustness of MAE. The basic idea is using the dataset-extracted domain knowledge to occupy the medium-/high-frequency of images, thus narrowing the optimization space of adversarial perturbations. Specifically, we group the distribution of pretraining data and optimize a set of cluster-specific visual prompts on frequency domain. These prompts are incorporated with input images through prototype-based prompt selection during test period. Extensive evaluation shows that our method clearly boost MAE’s adversarial robustness while maintaining its clean performance on ImageNet-1k classification. Our code is available at: https://github.com/shikiw/RobustMAE.

Hair editing has made tremendous progress in recent years. Early hair editing methods use well-drawn sketches or masks to specify the editing conditions. Even though they can enable very fine-grained local control, such interaction modes are inefficient for the editing conditions that can be easily specified by language descriptions or reference images. Thanks to the recent breakthrough of cross-modal models (e.g., CLIP), HairCLIP is the first work that enables hair editing based on text descriptions or reference images. However, such text-driven and reference-driven interaction modes make HairCLIP unable to support fine-grained controls specified by sketch or mask. In this paper, we propose HairCLIPv2, aiming to support all the aforementioned interactions with one unified framework. Simultaneously, it improves upon HairCLIP with better irrelevant attributes (e.g., identity, background) preservation and unseen text descriptions support. The key idea is to convert all the hair editing tasks into hair transfer tasks, with editing conditions converted into different proxies accordingly. The editing effects are added upon the input image by blending the corresponding proxy features within the hairstyle or hair color feature spaces. Besides the unprecedented user interaction mode support, quantitative and qualitative experiments demonstrate the superiority of HairCLIPv2 in terms of editing effects, irrelevant attribute preservation and visual naturalness. Our code is available at https://github.com/wty-ustc/HairCLIPv2.

In cloud storage applications, data owners’ original images are usually encrypted before being outsourced to the cloud for preserving data owners’ privacy. However, in deep learning model-based image encryption methods, an adversary can conduct the model extraction attack to reveal the model parameters and thus restore the privacy information by obtaining numerous encrypted images. In this paper, we propose an image translation-based deniable encryption (ITDE) scheme to achieve encryption deniability and defend against model extraction attacks. Differing from traditional encryption methods in which encrypted images are visually meaningless, ITDE applies image translation to generate encrypted images in the form of human faces. Moreover, ITDE provides deniability for data owners to keep the encryption parameters private. To defend against model extraction attacks, the defense mechanism is introduced in our proposed ITDE to preserve deep learning models. Experimental results demonstrate the superiority of our proposed methods in terms of encryption deniability and privacy preservation.

With the rapid development in media generation technologies, the creation of DeepFake videos is within everyone’s reach. As the widespread diffusion of DeepFakes can lead to severe consequences (e.g., defamation, fake news spreading, etc.), detecting DeepFakes is becoming a crucial task within the forensic community. However, most of the existing DeepFake detectors suffer from two issues: i) they are hardly explainable as they build upon black-box data-driven techniques rather than interpretable features; ii) they are often tailored to low-level texture features, failing to generalize on low-quality DeepFake videos. In this work we propose a video DeepFake detector that aims at solving these issues. The proposed detector relies on the fact that most DeepFake generators work on a frame-by-frame basis, thus breaking the temporal consistency of facial features across frames. In particular, we noticed that facial identity features tend to be less stable in time on DeepFake videos than original ones. We therefore propose a framework trained on time series of facial identity features. The use of high-level semantic features makes the detector interpretable and robust against low-quality DeepFake videos. Extensive experiments show that our method achieves outstanding performance on low-quality DeepFake video and obtains promising results on unseen dataset evaluation. The code is available at https://github.com/HongguLiu/Identity-Inconsistency-DeepFake-Detection

Infrared small target detection (ISTD) aims at segmenting the small targets from infrared images, which has wide applications in military surveillance. Present methods are mainly based on CNN and focus on modelling locality while ignoring global dependencies, which are indispensable because the local areas similar to small targets always spread over most of the background, causing heavy target ambiguity. Recently, RKformer [1] has combined local features with global dependencies and further introduced Runge-Kutta method, a one-step Ordinary Differential Equation (ODE) solver, to ISTD and performed well. However, the method simply fuses features from original transformer and residual blocks by naive concatenation, causing insufficient feature interaction. Also, it inevitably brings effective information loss, which greatly impairs ambiguous target features. To address above problems and target ambiguity, we introduce Adams-Bashforth-Moulton method and propose ABMNet, which has (1) multi-step memory and self-rectification mechanisms, guaranteeing more sufficient information usage and more accurate detection, (2) and achieves more sufficient interaction of both local and global information. Experiments on MDFA and IRSTD-1k demonstrate the superiority of our method.

Copy-Paste is a simple and effective data augmentation strategy for instance segmentation. By randomly pasting object instances onto new background images, it creates new training data for free and significantly boosts the segmentation performance, especially for rare object categories. Although diverse, high-quality object instances used in Copy-Paste result in more performance gain, previous works utilize object instances either from human-annotated instance segmentation datasets or rendered from 3D object models, and both approaches are too expensive to scale up to obtain good diversity. In this paper, we revisit Copy-Paste at scale with the power of newly emerged zero-shot recognition models (e.g., CLIP) and text2image models (e.g., StableDiffusion). We demonstrate for the first time that using a text2image model to generate images or zero-shot recognition model to filter noisily crawled images for different object categories is a feasible way to make Copy-Paste truly scalable. To make such success happen, we design a data acquisition and processing framework, dubbed ``X-Paste", upon which a systematic study is conducted. On the LVIS dataset, X-Paste provides impressive improvements over the strong baseline CenterNet2 with Swin-L as the backbone. Specifically, it archives +2.6 box AP and +2.1 mask AP gains on all classes and even more significant gains with +6.8 box AP +6.5 mask AP on long-tail classes.

Recent research has shown that neural network models can be used to steal sensitive data or embed malware. Therefore, steganalysis for neural networks is urgently needed. However, existing neural network steganalysis methods do not perform well under small embedding rates. In addition, because of the large number of parameters, the neural network steganography method under a small embedding rate can embed enough information into the model for malicious purposes. To address this problem, this paper proposes a calibration-based steganalysis method, which fine-tunes the original neural network model without implicit constraints to obtain a reference model, then extracts and fuses statistical moments from the parameter distributions of the original model and its reference model, and finally trains a logistic regressor for detection. Extensive experiments show that the proposed method has superior performance in detecting steganographic neural network models under small embedding rates.

Most infrared small target detection (ISTD) networks focus on building effective neural blocks or feature fusion modules but none describes the ISTD process from the image evolution perspective. The directional evolution of image pixels influenced by convolution, pooling and surrounding pixels is analogous to the movement of fluid elements constrained by surrounding variables ang particles. Inspired by this, we explore a novel research routine by abstracting the movement of pixels in the ISTD process as the flow of fluid in fluid dynamics (FD). Specifically, a new Fluid Dynamics-Inspired Network (FDI-Net) is devised for ISTD. Based on Taylor Central Difference (TCD) method, the TCD feature extraction block is designed, where convolution and Transformer structures are combined for local and global information. The pixel motion equation during the ISTD process is derived from the Navier–Stokes (N-S) equation, constructing a N-S Refinement Module that refines extracted features with edge details. Thus, the TCD feature extraction block determines the primary movement direction of pixels during detection, while the N-S Refinement Module corrects some skewed directions of the pixel stream to supplement the edge details. Experiments on IRSTD-1k and SIRST demonstrate that our method achieves SOTA performance in terms of evaluation metrics.

Online documents greatly improve the efficiency of information interaction but also cause potential security hazards, such as the ability to copy and reuse text content without authorization readily. To address copyright concerns, recent works have proposed converting reproducible text content into non-reproducible formats, making digital text content observable but not duplicable. However, as the Optical Character Recognition (OCR) technology develops, adversaries can still take screenshots of the target text region and use OCR to extract the text content. None of the existing methods can be well adapted to this kind of OCR extraction attack. In this paper, we propose "ProTegO'', a novel text content protection method against the OCR extraction attack, which generates adversarial underpaintings that do not affect human reading but can interfere with OCR after taking screenshots. Specifically, we design a text-style universal adversarial underpaintings generation framework, which can mislead both text recognition models and commercial OCR services. For invisibility, we take full advantage of the fusion property of human eyes and create complementary underpaintings to display alternatively on the screen. Experimental results demonstrate that ProTegO is a one-size-fits-all method that can ensure good visual quality while simultaneously achieving a high protection success rate on text recognition models with different architectures, outperforming the state-of-the-art methods. Furthermore, we validate the feasibility of ProTegO on a wide range of popular commercial OCR services, including Microsoft, Tencent, Alibaba, Huawei, Baidu, Apple, and Xiaomi. Codes will be available at https://github.com/Ruby-He/ProTegO.

The rapid progress of the DeepFake technique has caused severe privacy problems. Thus protecting facial data against DeepFake becomes an urgent requirement. Face protection can be regarded as a bidirectional process: Face-out-detection (FOD) and Face-in-forensics (FIF). For FOD, the detectability should be satisfied when using the protected face to replace other faces. For FIF, traceability should be guaranteed when the protected face is replaced by others. For this, we propose a Bidirectional Facial-data Protection Framework (BiFPro) to protect face data comprehensively. This framework is composed of three main parts: Watermarking embedding, Face-out-detection (FOD) and Face-in-forensics (FIF). For the FOD case, we ensure the vulnerability of the original face by embedding fragile watermarking. Once the protected facial image is used to replace other faces, the watermarking information will be corrupted in the synthesized face images which can be used to detect the authenticity of the protected facial images. As for the FIF case, we guarantee the traceability of the protected face image by embedding robust watermarking, with which the fake faces can be traced with the reserved watermarking even after the face is swapped. Experimental results demonstrate that our proposed BiFPro could generate the watermarking which is fragile to FOD and at the same time robust to FIF with an average watermark extraction success rate reaching more than 95% when defending against the four advanced DeepFake techniques. Finally, we hope this work can encourage more initiative countermeasures against DeepFake.

Deep 3D point cloud models are sensitive to adversarial attacks, which poses threats to safety-critical applications such as autonomous driving. Robust training and defend-by-denoising are typical strategies for defending adversarial perturbations. However, they either induce massive computational overhead or rely heavily upon specified priors, limiting generalized robustness against attacks of all kinds. To remedy it, this paper introduces a novel distortion-aware defense framework that can rebuild the pristine data distribution with a tailored intensity estimator and a diffusion model. To perform distortion-aware forward diffusion, we design a distortion estimation algorithm that is obtained by summing the distance of each point to the best-fitting plane of its local neighboring points, which is based on the observation of the local spatial properties of the adversarial point cloud. By iterative diffusion and reverse denoising, the perturbed point cloud under various distortions can be restored back to a clean distribution. This approach enables effective defense against adaptive attacks with varying noise budgets, enhancing the robustness of existing 3D deep recognition models.

Steganography is the act of disguising the transmission of secret information as seemingly innocent. Although provably secure steganography has been proposed for decades, it has not been mainstream in this field because its strict requirements (such as a perfect sampler and an explicit data distribution) are challenging to satisfy in traditional data environments. The popularity of deep generative models is gradually increasing and can provide an excellent opportunity to solve this problem. Several methods attempting to achieve provably secure steganography based on deep generative models have been proposed in recent years. However, they cannot achieve the expected security in practice due to unrealistic conditions, such as the balanced grouping of discrete elements and a perfect match between the message and channel distributions. In this paper, we propose a new provably secure steganography method in practice named Discop, which constructs several "distribution copies" during the generation process. At each time step of generation, the message determines from which "distribution copy" to sample. As long as the receiver agrees on some shared information with the sender, he can extract the message without error. To further improve the embedding rate, we recursively construct more "distribution copies" by creating Huffman trees. We prove that Discop can strictly maintain the original distribution so that the adversary cannot perform better than random guessing. Moreover, we conduct experiments on multiple generation tasks for diverse digital media, and the results show that Discop’s security and efficiency outperform those of previous methods.

Databases play a crucial role in storing and managing vast amounts of data in various organizations and industries. Yet the risk of database leakage poses a significant threat to data privacy and security. To trace the source of database leakage, researchers have proposed many database watermarking schemes. Among them, fake-tuples-based database watermarking shows great potential as it does not modify the original data of the database, ensuring the seamless usability of the watermarked database. However, the existing fake-tuple-based database watermarking schemes need to insert a large number of fake tuples for the embedding of each watermark bit, resulting in low watermark transparency. Therefore, we propose a novel database watermarking scheme based on fake tuples and sparse priority strategy, named SPSW, which achieves the same watermark capacity with a lower number of inserted fake tuples compared to the existing embedding strategy. Specifically, for a database about to be watermarked, we prioritize embedding the sparsest watermark sequence, i.e., the sequence containing the most ‘0’ bits among the currently available watermark sequences. For each bit in the sparse watermark sequence, when it is set to ‘1’, SPSW will embed the corresponding set of fake tuples into the database. Otherwise, no modifications will be made to the database. Through theoretical analysis, the proposed sparse priority strategy not only improves transparency but also enhances the robustness of the watermark. The comparative experimental results with other database watermarking schemes further validate the superior performance of the proposed SPSW, aligning with the theoretical analysis.

Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level or even instance-level multimodal results, still remains a challenge. In this article, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at both semantics and instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. To further extend the user interaction function of the proposed method, we also introduce sketches into the network. In addition, specially designed generator modules, Progressive Growing Module and Multi-Scale Refinement Module, can be used as a general module to improve the performance of complex scene generation. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Codes are available at https://github.com/tzt101/INADE.git.

An efficient predictor is essential in reversible data hiding methods. This paper proposes a video reversible data hiding method, where the correlations between pixels in both spatial and temporal domains are fully considered to improve prediction accuracy. In more detail, for each pixel, both intra prediction and inter prediction are implemented and the better prediction mode is selected according to intra or inter local complexity. A double-layered video partition scheme is applied so that the motion vectors generated during inter prediction are not necessarily embedded as additional information while reversibility is guaranteed. In addition, a cover pixel selection principle based on the probability distribution of the prediction errors is proposed, which prevents the pixels with large prediction errors from being modified. Experimental results demonstrate that the joint spatial and temporal prediction scheme offers satisfactory accuracy, and the proposed video reversible data hiding method can obtain high-quality stego videos.

Ternary AIFV codes are almost instantaneous fixed-to-variable length codes, and are constructed based on two code trees. It is known that the redundancy of ternary AIFV codes is no more than one. In this paper, we provide a tighter upper bound on the redundancy of the ternary Huffman codes when the greatest probability of the source$p_{max}$is known. As a result, the redundancy of the optimal ternary AIFV codes is bounded by Huffman codes, since the ternary Huffman codes can be seen as the special AIFV codes. To achieve lower redundancy than Huffman codes, we also propose a method to construct a class of ternary AIFV codes with time complexity$O(n)$for$n$source symbols. In addition, the redundancy of the proposed AIFV codes is analyzed and compared with Huffman codes. Analyzing the ternary AIFV codes constructed by the algorithm, we derive a tighter redundancy upper bounds under some conditions, which are superior to Huffman codes.

With the prosperity of online social networks (OSNs), people usually share photos taken with their mobile phones to OSN, which is suitable for covert communication. However, these images have high quality factors (QFs) and will be JPEG recompressed with low QFs by OSNs, which precludes existing robust steganography methods using low QF images. Therefore, we propose a Postprocessing and precise dither Modulation based robust Adaptive Steganography method (PMAS) for high quality images. Precise dither modulation ensures the robustness of the modified coefficients and improves security by reducing the magnitude of modifications. Postprocessing amends the coefficients that changed after recompression to ensure the robustness of unmodified coefficients. Additionally, we devise heuristics to mitigate post-processing and explore the relationship between the modification magnitude and distortion assignment with novel scaling functions, which all contribute to security. The experimental results demonstrate that PMAS is competent for high quality images, and the effect of every module in PMAS is verified.

For vacating-room-after-encryption reversible data hiding in encrypted images (VRAE RDHEI), an essential problem is how to address potential errors in data extraction and image recovery. This problem significantly limits the capacities of the existing VRAE RDHEI methods. To solve the problem while losing as little capacity as possible, in this paper, a novel method is proposed that uses the ideas of noisy channel model and channel code. By designing the data hiding mechanism appropriately, the embedding and extraction of data in the proposed method can be equivalent to the input and output of a virtual binary symmetric channel (BSC) model, so that the errors in data extraction are equivalent to the bit transitions in BSC. Based on the virtual BSC model, polar code is used to encode the secret data in the data hider's side. With the help of polar code, the receiver can decode the extracted bits containing errors to obtain correct secret data, then recover the error-free original image based on the corrected secret data. The experimental results proved that, compared with the existing VRAE methods, the proposed method can significantly improve the capacity and the quality of the decrypted images under the premise of complete reversibility.

Whereas cryptography easily arouses attacks by means of encrypting a secret message into a suspicious form, steganography is advantageous for its resilience to attacks by concealing the message in an innocent-looking cover signal. Minimal distortion steganography, one of the mainstream steganography frameworks, embeds messages while minimizing the distortion caused by the modification on the cover elements. Due to the unavailability of the original cover signal for the receiver, message embedding is realized by finding the coset leader of the syndrome function of steganographic codes migrated from channel coding, which is complex and has limited performance. Fortunately, deep generative models and the robust semantic of generated data make it possible for the receiver to perfectly reproduce the cover signal from the stego signal. With this advantage, we propose cover-reproducible steganography where the source coding, e.g., arithmetic coding, serves as the steganographic code. Specifically, the decoding process of arithmetic coding is used for message embedding and its encoding process is regarded as message extraction. Taking text-to-speech and text-to-image synthesis tasks as two examples, we illustrate the feasibility of cover-reproducible steganography. Steganalysis experiments and theoretical analysis are conducted to demonstrate that the proposed methods outperform the existing methods in most cases.

In recent years, face forgery detectors have aroused great interest and achieved impressive performance, but they are still struggling with generalization and robustness. In this work, we explore taking full advantage of the fine-grained forgery traces in both spatial and frequency domains to alleviate this issue. Specifically, we propose a novel High-Frequency Fine-Grained Transformer (F2Trans) network which contains two important components, namely Central Difference Attention (CDA) and High-frequency Wavelet Sampler (HWS). The premier CDA module is capable of capturing invariant fine-grained manipulation patterns by aggregating both pixel-level intensity and gradient information of the query to generate key and value pairs. Subsequently, the proposed HWS discards the low-frequency components of wavelet transformation and hierarchically explores high-frequency forgery cues of feature maps, which prevents model confusion caused by low-frequency components and pays attention to local frequency information. In addition, HWS can be employed as a special pooling layer for the F2Trans architecture to produce hierarchical feature representations in the spatial-frequency domain. Extensive experiments on multiple popular benchmarks demonstrate the generalization and robustness of the specially designed F2Trans framework is well-tailored for face forgery detection when confronting the cross-dataset, cross-manipulation, and unseen perturbations.

Nowadays, two-dimensional (2D) barcodes have been widely used in various domains. And a series of aesthetic 2D barcode schemes have been proposed to improve the visual quality and readability of 2D barcodes for better integration with marketing materials. Yet we believe that the existing aesthetic 2D barcode schemes arepartiallyaesthetic because they only beautify the data area but retain the position detection patterns with the blackwhite appearance of traditional 2D barcode schemes. Thus, in this paper, we propose the firstoverallaesthetic 2D barcode scheme, called OAcode, in which the position detection pattern is canceled. Its detection process is based on the pre-designed symmetrical data area of OAcode, whose symmetry could be used as the calibration signal to restore the perspective transformation in the barcode scanning process. Moreover, an enhanced demodulation method is proposed to resist the lens distortion common in the camera-shooting process. The experimental results illustrate that when 5×5cmOAcode is captured with a resolution of 720×1280 pixels, at the screen-camera distance of 10cmand the angle less or equal to 25°, OAcode has 100% detection rate and 99.5% demodulation accuracy. For 10×10cmOAcode, it could be extracted by consumer-grade mobile phones at a distance of 90cmwith around 90% accuracy.

Recent adversarial attack works attempt to improve the transferability by applying various differentiable transformations on input images. Considering the differentiable transformations and the original model together as a new model, these methods can be regarded as model augmentation that effectively derives an ensemble of models from the single original model. Despite their impressive performance, the model augmentation policies used in these methods are manually designed by experimental attempts, leaving the design of model augmentation policy an open question. In this paper, we propose an Automatic Model Augmentation (AutoMA) approach to find a strong model augmentation policy for transferable adversarial attacks. Specifically, we design a discrete search space that contains various diffierentiable transformations with different parameters and adopt reinforcement learning to search for the strong augmentation policy. The sampled augmentation policies together with the rewards they obtain during the searching process reveal several valuable observations for designing more powerful attacks using model augmentation policy:1) Augmentation transformations on color space are less effective; 2) The transformation type diversity matters; and 3) Using small distortion for geometric transformations while larger distortion for intensity transformations.Extensive experiments show that the augmentation policy found by AutoMA achieves superior performance than existing manually designed policies in a wide range of cases.

The quantum network is designed to connect numerous quantum nodes and support various ground-breaking quantum applications. Most of these applications require communicating parties to share entangled pairs. Therefore, entanglement routing, a technology distributing entangled pairs between distant quantum nodes, plays a vital role in realizing quantum networks’ capability. However, due to the limitation of quantum memory size and quantum decoherence, the entangled pairs shared by adjacent quantum nodes can hardly satisfy concurrent entanglement routing requests, thus leading to severe network congestion. In this paper, we propose a novel congestion mitigation (CM) scheme to tackle such bottleneck problems. The basic idea of CM is to “recycle” idle link-level entanglement resources from well-resourced links to bottleneck links utilizing a unique enabling technology of quantum networks, called entanglement swapping. CM can increase the capacity of each bottleneck link, thus overcoming resource limitations to improve resource utilization and network throughput. To complete our work, we also propose a swapping-based entanglement routing design, including path selection and resource allocation algorithms. Extensive simulations show that our design can significantly alleviate network congestion and improve the request service rate of quantum networks compared to the traditional entanglement routing designs.

In recent years, many model intellectual property (IP) proof methods for IP protection have been proposed, such as model watermarking and model fingerprinting. However, with the increasing number of models transmitted and deployed on the Internet, quickly finding the suspect model among thousands of models on model-sharing platforms such as GitHub is in great demand, which concurrently triggers the new security problem of model copy detection for IP protection. As an important part of the model IP protection system, the model copy detection task has not received enough attention. Due to the high computational complexity, both model watermarking and model fingerprinting lack the capability to efficiently find suspected infringing models among tens of millions of models. In this article, inspired by the hash-based image retrieval methods, we introduce a novel model copy detection mechanism: perceptual hashing for convolutional neural networks (CNNs). The proposed perceptual hashing algorithm can convert the weights of CNNs to fixed-length binary hash codes so that the lightly modified version has the similar hash code as the original model. By comparing the similarity of a pair of hash codes between a query model and a test model in the model library, similar versions of a query model can be retrieved efficiently. To the best of our knowledge, this is the first perceptual hashing algorithm for deep neural network models. Specifically, we first select the important model weights based on the model compression theory, then calculate the normal test statistics (NTS) on the segments of important weights, and finally encode the NTS features into hash codes. The experiment performed on a model library containing 3,565 models indicates that our perceptual hashing scheme has a superior copy detection performance.

Text content created by humans or language models is often stolen or misused by adversaries. Tracing text provenance can help claim the ownership of text content or identify the malicious users who distribute misleading content like machine-generated fake news. There have been some attempts to achieve this, mainly based on watermarking techniques. Specifically, traditional text watermarking methods embed watermarks by slightly altering text format like line spacing and font, which, however, are fragile to cross-media transmissions like OCR. Considering this, natural language watermarking methods represent watermarks by replacing words in original sentences with synonyms from handcrafted lexical resources (e.g., WordNet), but they do not consider the substitution’s impact on the overall sentence's meaning. Recently, a transformer-based network was proposed to embed watermarks by modifying the unobtrusive words (e.g., function words), which also impair the sentence's logical and semantic coherence. Besides, one well-trained network fails on other different types of text content. To address the limitations mentioned above, we propose a natural language watermarking scheme based on context-aware lexical substitution (LS). Specifically, we employ BERT to suggest LS candidates by inferring the semantic relatedness between the candidates and the original sentence. Based on this, a selection strategy in terms of synchronicity and substitutability is further designed to test whether a word is exactly suitable for carrying the watermark signal. Extensive experiments demonstrate that, under both objective and subjective metrics, our watermarking scheme can well preserve the semantic integrity of original sentences and has a better transferability than existing methods. Besides, the proposed LS approach outperforms the state-of-the-art approach on the Stanford Word Substitution Benchmark.

Bluetooth is a short-range wireless communication technology widely used by billions of personal computing, IoT, peripheral, and wearable devices. Bluetooth devices exchange commands and data, such as keyboard/mouse inputs, audio, and files, through a secure communication channel that is established through a pairing process. Due to the sensitivity of those commands and data, security mechanisms, such as encryption, authentication, and authorization, have been developed and adopted in the standards. Nevertheless, vulnerabilities continue to be discovered.

In this work we propose Identity Consistency Transformer, a novel face forgery detection method that focuses on high-level semantics, specifically identity information, and detecting a suspect face by finding identity inconsistency in inner and outer face regions. The Identity Consistency Transformer incorporates a consistency loss for identity consistency determination. We show that Identity Consistency Transformer exhibits superior generalization ability not only across different datasets but also across various types of image degradation forms found in real-world applications including deepfake videos. The Identity Consistency Transformer can be easily enhanced with additional identity information when such information is available, and for this reason it is especially well-suited for detecting face forgeries involving celebrities.11Code will be released at https://github.com/LightDXY/ICT_DeepFake

We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a mathematical analysis of the effect of the stripe width and vary the stripe width for different layers of the Transformer network which achieves strong modeling capability while limiting the computation cost. We also introduce Locally-enhanced Positional Encoding (LePE), which handles the local positional information better than existing encoding schemes. LePE naturally supports arbitrary input resolutions, and is thus especially effective and friendly for downstream tasks. Incorporated with these designs and a hierarchical structure, CSWin Transformer demonstrates competitive performance on common vision tasks. Specifically, it achieves 85.4% Top-1 accuracy on ImageNet-1K without any extra training data or label, 53.9 box AP and 46.4 mask AP on the COCO detection task, and 52.2 mIOU on the ADE20K semantic segmentation task, surpassing previous state-of-the-art Swin Transformer backbone by +1.2, +2.0, +1.4, and +2.0 respectively under the similar FLOPs setting. By further pretraining on the larger dataset ImageNet-21K, we achieve 87.5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55.7 mIoU.11Code and pretrain model is available at https://github.com/microsoft/CSWin-Transformer

Adversary and invisibility are two fundamental but conflict characters of adversarial perturbations. Previous adversarial attacks on 3D point cloud recognition have often been criticized for their noticeable point outliers, since they just involve an “implicit constrain” like global distance loss in the time-consuming optimization to limit the generated noise. While point cloud is a highly structured data format, it is hard to constrain its perturbation with a simple loss or metric properly. In this paper, we propose a novel Point-Cloud Sensitivity Map to boost both the efficiency and imperceptibility of point perturbations. This map reveals the vulnerability of point cloud recognition models when encountering shape-invariant adversarial noises. These noises are designed along the shape surface with an “explicit constrain” instead of extra distance loss. Specifically, we first apply a reversible coordinate transformation on each point of the point cloud input, to reduce one degree of point freedom and limit its movement on the tangent plane. Then we calculate the best attacking direction with the gradients of the transformed point cloud obtained on the white-box model. Finally we assign each point with a non-negative score to construct the sensitivity map, which benefits both white-box adversarial invisibility and black-box query-efficiency extended in our work. Extensive evaluations prove that our method can achieve the superior performance on various point cloud recognition models, with its satisfying adversarial imperceptibility and strong resistance to different point cloud defense settings. Our code is available at: https://github.com/shikiw/SI-Adv.

Transformers have achieved great success in pluralistic image inpainting recently. However, we find existing transformer based solutions regard each pixel as a token, thus suffer from information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration, incurring information loss and extra misalignment for the boundaries of masked regions. 2) They quantize 2563RGB pixels to a small number (such as 512) of quantized pixels. The indices of quantized pixels are used as tokens for the inputs and prediction targets of transformer. Although an extra CNN network is used to upsample and refine the low-resolution results, it is difficult to retrieve the lost information back. To keep input information as much as possible, we propose a new transformer based framework “PUT”. Specifically, to avoid input downsampling while maintaining the computation efficiency, we design a patch-based auto-encoder P-VQVAE, where the encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by quantization, an Un-Quantized Transformer (UQ-Transformer) is applied, which directly takes the features from P-VQVAE encoder as input without quantization and regards the quantized tokens only as prediction targets. Extensive experiments show that PUT greatly outperforms state-of-the-art methods on image fidelity, especially for large masked regions and complex large-scale datasets.

Hair editing is an interesting and challenging problem in computer vision and graphics. Many existing methods require well-drawn sketches or masks as conditional inputs for editing, however these interactions are neither straight-forward nor efficient. In order to free users from the tedious interaction process, this paper proposes a new hair editing interaction mode, which enables manipulating hair attributes individually or jointly based on the texts or reference images provided by users. For this purpose, we encode the image and text conditions in a shared embedding space and propose a unified hair editing framework by leveraging the powerful image text representation capability of the Contrastive Language-Image Pre-Training (CLIP) model. With the carefully designed network structures and loss functions, our framework can perform high-quality hair editing in a disentangled manner. Extensive experiments demonstrate the superiority of our approach in terms of manipulation accuracy, visual realism of editing results, and irrelevant attribute preservation.

The codebook is important for canonical Huffman coding, which needs to contain the number of leaves in each layer of the canonical Huffman tree and the corresponding symbols. Specifically, as two conventional methods in [1], [2], only the number of leaves in each level of the canonical Huffman tree is needed to store. However, we provide a new method to store the number of internal nodes in each layer and compactly encode the string of numbers according to the specific property between the internal nodes.

Collaborative learning in which local clients jointly train a deep learning model by sharing parameters to the central- ized server has gained great popularity. However, recent works have shown that local private data can be leaked to the server by gradient sharing. In this paper, a privacy-preserving collaborative learning scheme is proposed to defend against gradient-based reconstruction attacks. The sensitive training images are firstly permutated by transformation with scalable block sizes. Then, features of permutated images are extracted by a classification- compliant autoencoder for meaningful representation of high- dimensional images and facilitating classification. The model accuracy constraint is incorporated in the training process to maintain decent classification accuracy. Experimental results demonstrate that the proposed scheme can achieve high privacy preservation with minimal impact on model accuracy.

As an effective method for intellectual property (IP) protection, model watermarking technology has been applied on a wide variety of deep neural networks (DNN), including speech classification models. However, how to design a black-box watermarking scheme for automatic speech recognition (ASR) models is still an unsolved problem, which is a significant demand for protecting remote ASR Application Programming Interface (API) deployed in cloud servers. Due to conditional independence assumption and label-detection-based evasion attack risk of ASR models, the black-box model watermarking scheme for speech classification models cannot apply to ASR models. In this paper, we propose the first black-box model watermarking framework for protecting the IP of ASR models. Specifically, we synthesize trigger audios by spreading the speech clips of model owners over the entire input audios and labeling the trigger audios with the stego texts, which hides the authorship information with linguistic steganography. Experiments on the state-of-the-art open-source ASR system DeepSpeech demonstrate the feasibility of the proposed watermarking scheme, which is robust against five kinds of attacks and has little impact on accuracy.

With the development of social networks, traditional covert communication requires more consideration of lossy processes of Social Network Platforms (SNPs), which is called robust steganography. Since JPEG compression is a universal processing of SNPs, a method using repeated JPEG compression to fit transport channel matching is recently proposed and shows strong compression-resist performance. However, the repeated JPEG compression will inevitably introduce other artifacts into the stego image. Using only traditional steganalysis methods does not work well towards such robust steganography under low payload. In this paper, we propose a simple and effective method to detect the mentioned steganography by chasing both steganographic perturbations as well as continuous compression artifacts. We introduce compression-forensic features as a complement to steganalysis features, and then use the ensemble classifier for detection. Experiments demonstrate that this method owns a similar and better performance with respect to both traditional and neural-network-based steganalysis.

In recent years, the classification accuracy of CNN (convolutional neural network) steganalyzers has rapidly improved. However, as general CNN classifiers will misclassify adversarial samples, CNN steganalyzers can hardly detect adversarial steganography, which combines adversarial samples and steganography. Adversarial training and preprocessing are two effective methods to defend against adversarial samples. But literature shows adversarial training is ineffective for adversarial steganography. Steganographic modifications will also be destroyed by preprocessing, which aims to wipe out adversarial perturbations. In this paper, we propose a novel sampling based defense method for steganalysis. Specifically, by sampling image patches, CNN steganalyzers can bypass the sparse adversarial perturbations and extract effective features. Additionally, by calculating statistical vectors and regrouping deep features, the impact on the classification accuracy of common samples is effectively compressed. The experiments show that the proposed method can significantly improve the robustness against adversarial steganography without adversarial training.

Recently almost all the mainstream deepfake detection methods use Convolutional Neural Networks (CNN) as their backbone. However, due to the overreliance on local texture information which is usually determined by forgery methods of training data, these CNN-based methods cannot generalize well to unseen data. To get out of the predicament of prior methods, in this paper, we propose a novel transformer-based framework to model both global and local information and analyze anomalies of face images. In particular, we design attention leading module, multi-forensics module and variant residual connections for deepfake detection, and leverage token-level contrast loss for more detailed supervision. Experiments on almost all popular public deepfake datasets demonstrate that our method achieves state-of-the-art performance in cross-dataset evaluation and comparable performance in intra-dataset evaluation.

With the rapid development of general cloud services, more and more individuals or collectives use cloud platforms to store data. Assured data deletion deserves investigation in cloud storage. In time-sensitive data storage scenarios, it is necessary for cloud platforms to automatically destroy data after the data owner-specified expiration time. Therefore, assured time-sensitive data deletion should be sought. In this paper, a fine-grained assured time-sensitive data deletion (ATDD) scheme in cloud storage is proposed by embedding the time trapdoor in Ciphertext-Policy Attribute-Based Encryption (CP-ABE). Time-sensitive data is self-destructed after the data owner-specified expiration time so that the authorized users cannot get access to the related data. In addition, a credential is returned to the data owner for data deletion verification. This proposed scheme provides solutions for fine-grained access control and verifiable data self-destruction. Detailed security and performance analysis demonstrate the security and the practicability of the proposed scheme.

With the popularization of online services, a lot of text watermarking algorithm have been proposed to protect the digital documents. However, most of them require extra manual design or text samantic modification. This paper proposes an end-to-end font watermarking network which is capable of automatic watermark embedding and extraction without changing the text content. In our scheme, a watermark is generated by changing the font attributes slightly and then embedded as a tiny perturbation. And we extract the watermark by detecting the attribute values of the font image. Experimental results highlight the superiority of the proposed watermarking scheme in terms of imperceptibility and speed comparing to the existing work.

Existing face forgery detection methods usually treat face forgery detection as a binary classification problem and adopt deep convolution neural networks to learn discriminative features. The ideal discriminative features should be only related to the real/fake labels of facial images. However, we observe that the features learned by vanilla classification networks are correlated to unnecessary properties, such as forgery methods and facial identities. Such phenomenon would limit forgery detection performance especially for the generalization ability. Motivated by this, we propose a novel method which utilizes adversarial learning to eliminate the negative effect of different forgery methods and facial identities, which helps classification network to learn intrinsic common discriminative features for face forgery detection. To leverage data lacking ground truth label of facial identities, we design a special identity discriminator based on similarity information derived from off-the-shelf face recognition model. Extensive experiments demonstrate the effectiveness of the proposed method under both intra-dataset and cross-dataset evaluation settings.

Federated learning allows multiple clients to collab-oratively train a global deep learning model without revealing their local data to a centralized server. However, the existence of clients whose datasets have imbalanced class distribution has a significant impact on model accuracy. Imbalance makes it challenging for a model to distinguish between the majority and minority classes without accessing clients' local data. In this paper, we aim to tackle this problem by privacy-preserving cost-sensitive federated learning. We design the joint cost-sensitive and differentially private model parameter optimization mechanism which maintains the model accuracy while satisfying differential privacy constraints. Moreover, this mechanism does not alter the original data distribution. Experimental results demonstrate the superior performance of our proposed scheme in terms of model accuracy and privacy preservation.

Due to the increasing popularity of cloud computing and privacy preservation concerns, sensitive data should be encrypted before outsourcing to the cloud and data utilization becomes a challenging issue. Searchable encryption (SE) is a promising technique to address this problem. Most existing searchable encryption schemes only support accurate keyword but fuzzy keyword search schemes are appreciated in practice. Moreover, in some application scenarios like the video on demand systems, data owners only hope to give the access of their data to those who have payed but authenticated user identity may often change. Therefore, dynamic user attribute updating should be considered. To solve about issues, we propose a fuzzy secure keyword search scheme over encrypted cloud data with dynamic fine-grained access control. We design a novel fuzzy keyword index to retrieve corresponding documents. To reduce the computation cost, the cloud server selects most relevant top-k documents and return them to the data users. The ciphertext-policy attribute based encryption (CP-ABE) technique is introduced to implement fine-grained access control. Meanwhile, the basic CP-ABE is improved to meet the practical need of user attribute updating. Extensive security and performance analysis demonstrates that our proposed scheme is highly efficient and can satisfy the security requirements for fuzzy keyword search over encrypted cloud data.

Abstract Robust 3D mesh watermarking is a traditional research topic in computer graphics, which provides an efficient solution to the copyright protection for 3D meshes. Traditionally, researchers need manually design watermarking algorithms to achieve sufficient robustness for the actual application scenarios. In this paper, we propose the first deep learning-based 3D mesh watermarking network, which can provide a more general framework for this problem. In detail, we propose an end-to-end network, consisting of a watermark embedding sub-network, a watermark extracting sub-network and attack layers. We employ the topology-agnostic graph convolutional network (GCN) as the basic convolution operation, therefore our network is not limited by registered meshes (which share a fixed topology). For the specific application scenario, we can integrate the corresponding attack layers to guarantee adaptive robustness against possible attacks. To ensure the visual quality of watermarked 3D meshes, we design the curvature consistency loss function to constrain the local geometry smoothness of watermarked meshes. Experimental results show that the proposed method can achieve more universal robustness while guaranteeing comparable visual quality.

Spatially-adaptive normalization (SPADE) is remarkably successful recently in conditional semantic image synthesis in T. Park et al. 2019 which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to prevent the semantic information from being washed away. Despite its impressive performance, a more thorough understanding of the advantages inside the box is still highly demanded to help reduce the significant computation and parameter overhead introduced by this novel structure. In this paper, from a return-on-investment point of view, we conduct an in-depth analysis of the effectiveness of this spatially-adaptive normalization and observe that its modulation parameters benefit more from semantic-awareness rather than spatial-adaptiveness, especially for high-resolution input masks. Inspired by this observation, we propose class-adaptive normalization (CLADE), a lightweight but equally-effective variant that is only adaptive to semantic class. In order to further improve spatial-adaptiveness, we introduce intra-class positional map encoding calculated from semantic layouts to modulate the normalization parameters of CLADE and propose a truly spatially-adaptive variant of CLADE, namely CLADE-ICPE. Through extensive experiments on multiple challenging datasets, we demonstrate that the proposed CLADE can be generalized to different SPADE-based methods while achieving comparable generation quality compared to SPADE, but it is much more efficient with fewer extra parameters and lower computational cost. The code and pretrained models are available at https://github.com/tzt101/CLADE.git.

Despite the tremendous success, deep neural networks are exposed to serious IP infringement risks. Given a target deep model, if the attacker knows its full information, it can be easily stolen by fine-tuning. Even if only its output is accessible, a surrogate model can be trained through student-teacher learning by generating many input-output training pairs. Therefore, deep model IP protection is important and necessary. However, it is still seriously under-researched. In this work, we propose a new model watermarking framework for protecting deep networks trained for low-level computer vision or image processing tasks. Specifically, a special task-agnostic barrier is added after the target model, which embeds a unified and invisible watermark into its outputs. When the attacker trains one surrogate model by using the input-output pairs of the barrier target model, the hidden watermark will be learned and extracted afterwards. To enable watermarks from binary bits to high-resolution images, a deep invisible watermarking mechanism is designed. By jointly training the target model and watermark embedding, the extra barrier can even be absorbed into the target model. Through extensive experiments, we demonstrate the robustness of the proposed framework, which can resist attacks with different network structures and objective functions.

Remote entanglement distribution plays a crucial role in large-scale quantum networks, and the key enabler for entanglement distribution is quantum routers (or repeaters) that can extend the entanglement transmission distance. However, the performance of quantum routers is far from perfect yet. Amongst the causes, the limited quantum memories in quantum routers largely affect the rate and efficiency of entanglement distribution. To overcome this challenge, this paper presents a new modeling for the maximization of entanglement distribution rate (EDR) on a memory-limited path, which is then transformed into entanglement generation and swapping sub-problems. We propose a greedy algorithm for short-distance entanglement generation so that the quantum memories can be efficiently used. As for the entanglement swapping sub-problem, we model it using an Entanglement Graph (EG), whose solution is yet found to be at least NP-complete. In light of it, we propose a heuristic algorithm by dividing the original EG into several sub-problems, each of which can be solved using dynamic programming (DP) in polynomial time. By conducting simulations, the results show that our proposed scheme can achieve a high EDR, and the developed algorithm has a polynomial-time upper bound and reasonable average runtime complexity.

Entanglement routing establishes remote entanglement connection between two arbitrary nodes, which is one of the most important functions in quantum networks. The existing routing mechanisms mainly improve the robustness and throughput facing the failure of entanglement generations, which, however, rarely include the considerations on the most important metric to evaluate the quality of connection, entanglement fidelity. To solve this problem, we propose purification-enabled entanglement routing designs to provide fidelity guarantee for multiple Source-Destination (S-D) pairs in quantum networks. In our proposal, we first consider the single S-D pair scenario and design an iterative routing algorithm, Q-PATH, to find the optimal purification decisions along the routing path with minimum entangled pair cost. Further, a low-complexity routing algorithm using an extended Dijkstra algorithm, Q-LEAP, is designed to reduce the computational complexity by using a simple but effective purification decision method. Finally, we consider the common scenario with multiple S-D pairs and design a greedy-based algorithm considering resource allocation and re-routing process for multiple routing requests. Simulation results show that the proposed algorithms not only can provide fidelity-guaranteed routing solutions, but also has superior performance in terms of throughput, fidelity of end-to-end entanglement connection, and resource utilization ratio, compared with the existing routing scheme.

JPEG steganalysis aims to detect stego JPEG images. For some robust steganography methods, in order to enhance stego images’ robustness of resisting JPEG recompression from lossy channel such SNS or photo sharing websites, steganographer may intentionally recompress cover image several times with quantization matrix of targeted channel, which thereby make it possible to transmit stego data in such channel for better disguise. In addition, there are huge number of cover JPEG images may be recompressed for various reasons, such as processing by some tools. Thus a better steganalysis method for such images is needed. In this paper, we investigate the steganalysis method for images recompressed with the same quantization matrix, namely, discriminate recompressed JPEG cover images and its stego images. We present some observed phenomenon on recompressed JPEG images, and design methods to enhance the sensitivity of feature based and deep model based steganalysis methods for this task. To verify their effectiveness with different acquisition of recompression prior-knowledge, we conduct experiments in various settings including conventional setting and mixing samples of different recompressing times in training. Their results demonstrate that the proposed method can notably improve detection accuracy on recompressed JPEG images.

Steganography is the art and science of hiding secret messages in public communication so that the presence of secret messages cannot be detected. There are two distribution-preserving steganographic frameworks, one is sampler-based and the other is compression-based. The former requires a perfect sampler which yields data following the same distribution, and the latter needs the explicit distribution of generative objects. However, these two conditions are too strict even unrealistic in the traditional data environment, e.g., the distribution of natural images is hard to seize. Fortunately, generative models bring new vitality to distribution-preserving steganography, which can serve as the perfect sampler or provide the explicit distribution of generative media. Taking text-to-speech generation task as an example, we propose distribution-preserving steganography based on WaveGlow and WaveRNN, which corresponds to the former two categories. Steganalysis experiments and theoretical analysis are conducted to demonstrate that the proposed methods can preserve the distribution.

Under the assumption of honest-but-curious cloud service provider, various cryptographic techniques have been used to address the issues of data access control and confidentiality in public cloud storage. Among which, attribute-based encryption (ABE) has been shown to be an attractive scheme. Although the technique of ABE brings in various benefits, its onerous overhead should not be ignored. In this article, based on an improved LSSS (linear secret sharing scheme) matrix expression integrated in CP-ABE (Ciphertext-Policy Attribute-Based Encryption) algorithm, we present an efficient and secure attribute-based access control scheme for the scenarios where multiple data are shared and encrypted with frequently used sub-policies. In the scheme, a user can store the parameters about a specific sub-policy in his/her first decryption, which can be reused in the subsequent data decryptions whose embedded access policies include the same sub-policy so as to significantly reduce the computation cost. Our proposed scheme is proved to be semantically secure under chosen plaintext attacks and can well preserve the confidentiality of the data sharing system. Our analysis and experimentation also show that our scheme does significantly reduce the decryption time and while trades in only very little storage overhead, and thus effectively promotes the efficiency.

Recently, face manipulation techniques have caused increasing trust concerns in our society. Although current face manipulation detection methods achieve impressive performance regarding intra-dataset evaluation, they are struggling to improve the generalization and robustness ability. To address this issue, we propose a novel Hierarchical Frequency-assisted Interactive Networks (HFI-Net) to explore comprehensive frequency-related forgery cues for face manipulation detection. At first, we formulate HFI-Net as a dual-branch network to take full advantage of both CNN and transformer for capturing local details and global context information, respectively. Considering the forged faces are easy to show flaws in the frequency domain, a novel Frequency-based Feature Refinement (FFR) module is proposed to learn frequency-based attention from RGB features. FFR module emphasizes forgery cues and suppresses the pristine semantics information by keeping middle-high frequency features while discarding the low-frequency ones. Based on FFR, we further develop a co-sharing Global-Local Interaction (GLI) module to conduct frequency-assisted interactions while capturing complementarity among dual branches. Lastly, we further implement the GLI module in each stage of the network to effectively explore multi-level frequency artifacts. Extensive experiments are conducted on several popular benchmarks including FaceForensics++, Celeb-DF, DeepFake-TIMIT, DFDC, UADFV, and DeeperForensics-1.0, which shows that our model outperforms the state-of-the-art, especially in unseen datasets, manipulations, and perturbations evaluation.

This paper studies the problem of StyleGAN inversion, which plays an essential role in enabling the pretrained StyleGAN to be used for real image editing tasks. The goal of StyleGAN inversion is to find the exact latent code of the given image in the latent space of StyleGAN. This problem has a high demand for quality and efficiency. Existing optimization-based methods can produce high-quality results, but the optimization often takes a long time. On the contrary, forward-based methods are usually faster but the quality of their results is inferior. In this paper, we present a new feed-forward network "E2Style" for StyleGAN inversion, with significant improvement in terms of efficiency and effectiveness. In our inversion network, we introduce: 1) a shallower backbone with multiple efficient heads across scales; 2) multi-layer identity loss and multi-layer face parsing loss to the loss function; and 3) multi-stage refinement. Combining these designs together forms an effective and efficient method that exploits all benefits of optimization-based and forward-based methods. Quantitative and qualitative results show that our E2Style performs better than existing forward-based methods and comparably to state-of-the-art optimization-based methods while maintaining the high efficiency as well as forward-based methods. Moreover, a number of real image editing applications demonstrate the efficacy of our E2Style. Our code is available at https://github.com/wty-ustc/e2style.

Recent research shows deep neural networks are vulnerable to different types of attacks, such as adversarial attacks, data poisoning attacks, and backdoor attacks. Among them, backdoor attacks are the most cunning and can occur in almost every stage of the deep learning pipeline. Backdoor attacks have attracted lots of interest from both academia and industry. However, most existing backdoor attack methods are visible or fragile to some effortless pre-processing such as common data transformations. To address these limitations, we propose a robust and invisible backdoor attack called "Poison Ink". Concretely, we first leverage the image structures as target poisoning areas and fill them with poison ink (information) to generate the trigger pattern. As the image structure can keep its semantic meaning during the data transformation, such a trigger pattern is inherently robust to data transformations. Then we leverage a deep injection network to embed such input-aware trigger pattern into the cover image to achieve stealthiness. Compared to existing popular backdoor attack methods, Poison Ink outperforms both in stealthiness and robustness. Through extensive experiments, we demonstrate that Poison Ink is not only general to different datasets and network architectures but also flexible for different attack scenarios. Besides, it also has very strong resistance against many state-of-the-art defense techniques.

With the rapid development of digital devices, the issue of how to transmit information among different devices with multimedia carriers has drawn much attention from the research community. This paper focuses on the important user scenario of “screen-to-camera information transmission”. Along this direction, image coding-based techniques have been shown to be the most popular and effective methods in the past decades. However, after careful study, we find that none of the existing methods can satisfy the four important properties simultaneously, i.e.,high transparency,high embedding efficiency,strong transmission robustnessandhigh adaptability to device types. This is mainly because these properties are contradictory with each other. In this paper, we thus propose a screen-to-camera image code dubbed “TERA” (transparency,efficiency,robustness andadaptability), which makes it possible to circumvent the contradiction among the above four properties for the first time. Generally, TERA adopts the color decomposition principle to ensure the visual quality and the superposition-based scheme to ensure embedding efficiency. BCH-coding-based information arrangement and a powerful attention-guided information decoding network are further designed to guarantee the robustness and adaptability. Through extensive experiments, the superiority and broad applications of our method are demonstrated.

In recent years, deep learning-based steganalyzers far outperformed handcrafted feature-based steganalyzers. However, a large amount of data is needed to train deep learning networks. For steganalysis tasks, the steganographic traces are subtle and the steganographic signals are difficult to be captured when the number of cover/stego pairs in the training set is insufficient. Data augmentation has been proved to be effective in improving accuracy and generalization for deep learning models. Yet not all data augmentation methods are universal for all tasks. When performing data augmentation, we argue that data distribution under the target tasks should be maintained. Since the steganalysis task is more concerned with the high-frequency signals of the images, if the high-frequency signals are unchanged, the data distribution from the perspective of steganalysis will remain largely unchanged. Based on this principle, we designed a neural network called cover augmentation network, which enriches the dataset by intelligently adding noise to the original cover to generate the augmented cover. Further, we designed a whole process of data augmentation based on the cover augmentation network. Experimental results show that the proposed data augmentation method can effectively improve the performance of steganalysis networks, and the advantage is significant at low payloads.

In this article, we propose secure plaintext image storage protocols in the cloud environment for image owners managing and controlling their outsourced images. To solve control and privacy issues, conventional schemes suggest outsourcing images in the encrypted form. However, encrypted images lose their usability and visibility. For example, image owners cannot quickly find their stored images in the cloud. The proposed protocols encourage plaintext image storage in the cloud with copyright protection of images, visible display, lossless retrieval and controlled deletion via techniques of homomorphic encryption and digital watermarking. We allow the image owner to embed and remove the watermark in a privacy-preserving way without compromising the security of the original data and the computed results. Moreover, the image owner can securely detect the cloud's dishonest act of not completely deleting an image as required or leaking an image without permission. Compared with existing works, our work achieves more functions. We prove that the proposed work achieves the controllable management of the outsourced plaintext images without privacy leakage to unauthorized parties and demonstrate the utility and the efficiency of our protocols through experimental evaluation.

Invertible grayscale is a special kind of grayscale from which the original color can be recovered. Given an input color image, this seminal work tries to hide the color information into its grayscale counterpart while making it hard to recognize any anomalies. This powerful functionality is enabled by training a hiding sub-network and restoring sub-network in an end-to-end way. Despite its expressive results, two key limitations exist: 1) The restored color image often suffers from some noticeable visual artifacts in the smooth regions. 2) It is very sensitive to JPEG compression, i.e., the original color information cannot be well recovered once the intermediate grayscale image is compressed by JPEG. To overcome these two limitations, this article introduces adversarial training and JPEG simulator respectively. Specifically, two auxiliary adversarial networks are incorporated to make the intermediate grayscale images and final restored color images indistinguishable from normal grayscale and color images. And the JPEG simulator is utilized to simulate real JPEG compression during the online training so that the hiding and restoring sub-networks can automatically learn to be JPEG robust. Extensive experiments demonstrate that the proposed method is superior to the original invertible grayscale work both qualitatively and quantitatively while ensuring the JPEG robustness. We further show that the proposed framework can be applied under different types of grayscale constraints and achieve excellent results.

Three-dimensional (3-D) meshes are commonly used to represent virtual surfaces and volumes. Over the past decade, 3-D meshes have emerged in industrial, medical, and entertainment applications, being of large practical significance for 3-D mesh steganography and steganalysis. In this article, we provide a systematic survey of the literature on 3-D mesh steganography and steganalysis. Compared with an earlier survey (Girdhar et al., 2017), we propose a new taxonomy of steganographic algorithms with four categories: 1) two-state domain, 2) LSB domain, 3) permutation domain, and 4) transform domain. Regarding steganalysis algorithms, we divide them into two categories: 1) universal steganalysis and 2) specific steganalysis. For each category, the history of technical developments and the current technological level are introduced and discussed. Finally, we highlight some promising future research directions and challenges in improving the performance of 3-D mesh steganography and steganalysis.

Video object detection is challenging in the presence of appearance deterioration in certain video frames. Therefore, it is a natural choice to aggregate temporal information from other frames of the same video into the current frame. However, ROI Align, as one of the most core procedures of video detectors, still remains extracting features from a single-frame feature map for proposals, making the extracted ROI features lack temporal information from videos. In this work, considering the features of the same object instance are highly similar among frames in a video, a novel Temporal ROI Align operator is proposed to extract features from other frames feature maps for current frame proposals by utilizing feature similarity. The proposed Temporal ROI Align operator can extract temporal information from the entire video for proposals. We integrate it into single-frame video detectors and other state-of-the-art video detectors, and conduct quantitative experiments to demonstrate that the proposed Temporal ROI Align operator can consistently and significantly boost the performance. Besides, the proposed Temporal ROI Align can also be applied into video instance segmentation.

Benefiting from the development of generative adversarial networks (GAN), facial manipulation has achieved significant progress in both academia and industry recently. It inspires an increasing number of entertainment applications but also incurs severe threats to individual privacy and even political security meanwhile. To mitigate such risks, many countermeasures have been proposed. However, the great majority methods are designed in a passive manner, which is to detect whether the facial images or videos are tampered after their wide propagation. These detection-based methods have a fatal limitation, that is, they only work for ex-post forensics but can not prevent the engendering of malicious behavior. To address the limitation, in this paper, we propose a novel framework of initiative defense to degrade the performance of facial manipulation models controlled by malicious users. The basic idea is to actively inject imperceptible venom into target facial data before manipulation. To this end, we first imitate the target manipulation model with a surrogate model, and then devise a poison perturbation generator to obtain the desired venom. An alternating training strategy are further leveraged to train both the surrogate model and the perturbation generator. Two typical facial manipulation tasks: face attribute editing and face reenactment, are considered in our initiative defense framework. Extensive experiments demonstrate the effectiveness and robustness of our framework in different settings. Finally, we hope this work can shed some light on initiative countermeasures against more adversarial scenarios.

Visible-infrared cross modality person re-identification (VI-ReID) is a core but challenging technology in the 24-hours intelligent surveillance system. How to eliminate the large modality gap lies in the heart of VI-ReID. Conventional methods mainly focus on directly aligning the heterogeneous modalities into the same space. However, due to the unbalanced color information between the visible and infrared images, the features of visible images tend to overfit the clothing color information, which would be harmful to the modality alignment. Besides, these methods mainly align the heterogeneous feature distributions in dataset-level while ignoring the valuable identity information, which may cause the feature misalignment of some identities and weaken the discrimination of features. To tackle above problems, we propose a novel approach for VI-ReID. It learns the color-irrelevant features through the color-irrelevant consistency learning (CICL) and aligns the identity-level feature distributions by the identity-aware modality adaptation (IAMA). The CICL and IAMA are integrated into a joint learning framework and can promote each other. Extensive experiments on two popular datasets SYSU-MM01 and RegDB demonstrate the superiority and effectiveness of our approach against the state-of-the-art methods.

The remarkable success in face forgery techniques has received considerable attention in computer vision due to security concerns. We observe that up-sampling is a necessary step of most face forgery techniques, and cumulative up-sampling will result in obvious changes in the frequency domain, especially in the phase spectrum. According to the property of natural images, the phase spectrum preserves abundant frequency components that provide extra information and complement the loss of the amplitude spectrum. To this end, we present a novel Spatial-Phase Shallow Learning (SPSL) method, which combines spatial image and phase spectrum to capture the up-sampling artifacts of face forgery to improve the transferability, for face forgery detection. And we also theoretically analyze the validity of utilizing the phase spectrum. Moreover, we notice that local texture information is more crucial than high-level semantic information for the face forgery detection task. So we reduce the receptive fields by shallowing the network to suppress high-level features and focus on the local region. Extensive experiments show that SPSL can achieve the state-of-the-art performance on cross-datasets evaluation as well as multi-class classification and obtain comparable results on single dataset evaluation.

Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level multimodal results, still remains a challenge. In this paper, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at semantic or even instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Code will be available at https://github.com/tzt101/INADE.git

Image matting is a fundamental and challenging problem in computer vision and graphics. Most existing matting methods leverage a user-supplied trimap as an auxiliary input to produce good alpha matte. However, obtaining high-quality trimap itself is arduous, thus restricting the application of these methods. Recently, some trimap-free methods have emerged, however, the matting quality is still far behind the trimap-based methods. The main reason is that, without the trimap guidance in some cases, the target network is ambiguous about which is the foreground target. In fact, choosing the foreground is a subjective procedure and depends on the user’s intention. To this end, this paper proposes an improved deep image matting framework which is trimap-free and only needs several user click interactions to eliminate the ambiguity. Moreover, we introduce a new uncertainty estimation module that can predict which parts need polishing and a following local refinement module. Based on the computation budget, users can choose how many local parts to improve with the uncertainty guidance. Quantitative and qualitative results show that our method performs better than existing trimap-free methods and comparably to state-of-the-art trimap-based methods with minimal user effort.

Face forgery by deepfake is widely spread over the internet and has raised severe societal concerns. Recently, how to detect such forgery contents has become a hot research topic and many deepfake detection methods have been proposed. Most of them model deepfake detection as a vanilla binary classification problem, i.e, first use a backbone network to extract a global feature and then feed it into a binary classifier (real/fake). But since the difference between the real and fake images in this task is often subtle and local, we argue this vanilla solution is not optimal. In this paper, we instead formulate deepfake detection as a fine-grained classification problem and propose a new multi-attentional deepfake detection network. Specifically, it consists of three key components: 1) multiple spatial attention heads to make the network attend to different local parts; 2) textural feature enhancement block to zoom in the subtle artifacts in shallow features; 3) aggregate the low-level textural feature and high-level semantic features guided by the attention maps. Moreover, to address the learning difficulty of this network, we further introduce a new regional independence loss and an attention guided data augmentation strategy. Through extensive experiments on different datasets, we demonstrate the superiority of our method over the vanilla binary classifier counterparts, and achieve state-of-the-art performance. The models will be released recently at https://github.com/yoctta/multiple-attention.

Deep neural networks have been proved that they are vulnerable to adversarial examples, which are generated by adding human-imperceptible perturbations to images. To defend these adversarial examples, various detection based methods have been proposed. However, most of them perform poorly on detecting adversarial examples with extremely slight perturbations. By exploring these adversarial examples, we find that there exists compliance between perturbations and prediction confidence, which guides us to detect few-perturbation attacks from the aspect of prediction confidence. To detect both few-perturbation attacks and large-perturbation attacks, we propose a method beyond image space by a two-stream architecture, in which the image stream focuses on the pixel artifacts and the gradient stream copes with the confidence artifacts. The experimental results show that the proposed method outperforms the existing methods under oblivious attacks and is verified effective to defend omniscient attacks as well.

Co-occurrent visual pattern makes aggregating contextual information a common paradigm to enhance the pixel representation for semantic image segmentation. The existing approaches focus on modeling the context from the perspective of the whole image, i.e., aggregating the image-level contextual information. Despite impressive, these methods weaken the significance of the pixel representations of the same category, i.e., the semantic-level contextual information. To address this, this paper proposes to augment the pixel representations by aggregating the image-level and semantic-level contextual information, respectively. First, an image-level context module is designed to capture the contextual information for each pixel in the whole image. Second, we aggregate the representations of the same category for each pixel where the category regions are learned under the supervision of the ground-truth segmentation. Third, we compute the similarities between each pixel representation and the image-level contextual information, the semantic-level contextual information, respectively. At last, a pixel representation is augmented by weighted aggregating both the image-level contextual information and the semantic-level contextual information with the similarities as the weights. Integrating the image-level and semantic-level context allows this paper to report state-of-the-art accuracy on four benchmarks, i.e., ADE20K, LIP, COCOStuff and Cityscapes1.

Unsupervised pretraining has achieved great success and many recent works have shown unsupervised pretraining can achieve comparable or even slightly better transfer performance than supervised pretraining on downstream target datasets. But in this paper, we find this conclusion may not hold when the target dataset has very few labeled samples for finetuning, i.e., few-label transfer. We analyze the possible reason from the clustering perspective: 1) The clustering quality of target samples is of great importance to few-label transfer; 2) Though contrastive learning is essential to learn how to cluster, its clustering quality is still inferior to supervised pretraining due to lack of label supervision. Based on the analysis, we interestingly discover that only involving some unlabeled target domain into the unsupervised pretraining can improve the clustering quality, subsequently reducing the transfer performance gap with supervised pretraining. This finding also motivates us to propose a new progressive few-label transfer algorithm for real applications, which aims to maximize the transfer performance under a limited annotation budget. To support our analysis and proposed method, we conduct extensive experiments on nine different target datasets. Experimental results show our proposed method can significantly boost the few-label transfer performance of unsupervised pretraining.

The current spike of deepfake techniques has received considerable attention due to security concerns. To mitigate the potential risks brought by deepfake techniques, many detection methods have been proposed. However, most existing works merely leverage spatial information from separate frames and ignore valuable inter-frame temporal information. In this paper, we propose a deepfake detection scheme that uses 3D-attentional inception network. The proposed model encompasses both spatial and temporal information simultaneously with the 3D kernels. Furthermore, the channel and spatial-temporal attention modules are applied to improve detection capabilities. Comprehensive experiments demonstrate that our scheme outperforms state-of-the-art methods.

To transfer head pose and expression with few photographs is a novel yet challenging task in deepfake generation. Despite impressive results have been achieved in related works, there are still two limitations in the existing methods: 1) most of the methods are based on computer graphics, which take a lot of computing resources, while lacking of generalization for different identity, 2) few-shot based methods cannot handle the few-shot style transfer video generation. To address these distortion problems, we propose a novel deep learning framework, named as Few-Shot Face Transfer Networks(FSFT-Net) which works for the face transfer video generation. The proposed FSFT-Net driven by arbitrary portrait video involves a cascaded-based style generator to synthesize stable video with few free-view images. In addition, the frame and video discriminators are adopted for optimization of the proposed generator. The FSFT-Net performs long-term adversarial training on large-scale video datasets. Extensive experiments demonstrate that our FSFT-Net outperforms state-of-the-art methods both quantitatively and qualitatively results.

User identity authentication is essencial for ensuring information security. With the widespread use of electronic devices, online handwriting verification becomes more important in identity authentication based on biometrics and widely used in financial, commercial, and forensic fields. In this paper, we propose a multi-path feature fusion network for multi-modal fusion of static and dynamic handwriting obtained by electronic devices to intensify the handwriting verification. Since traditional handwritten signature verification, of which the handwritten content just the writer’s name, is vulnerable to skilled forgery attacks, we propose a content-independent handwriting verification scheme to solve this problem. We also build a handwriting dataset with approximately 5400 samples of 30 individuals’ handwriting, which contributes to extracting content-independent handwriting style features. We test our method on widely used BiosecurID dataset and our dataset. The experimental results demonstrate the feasibility of the proposed method.

Different from close-set classification task, deep face recognition models are often used in open-set scenarios, where the models need to handle arbitrary faces. Open-set adversarial attacks can identify the vulnerability of deep face recognition models. Compared to time-consuming iterative gradient-based methods, generator-based methods can produce adversarial examples with only one forward pass, which greatly improves attack efficiency. However, existing generator-based attack methods need to train an individual model for each target identity and can only generate a fixed perturbation pattern regardless of different attack intensity constraints, which is impractical and sub-optimal for open-set adversarial attacks. In this paper, we propose an efficient generator-based Single Model ARbitrary Target (SMART) approach for open-set adversarial attacks against deep face recognition models. Given an arbitrary source-target face image pair, SMART first generates an additive perturbation and then adds it to the source image to obtain the final adversarial face image. After the training with various source-target pairs randomly sampled on large scale face images, SMART could effectively learn inherent perturbation patterns for arbitrary source-target face images pairs. Besides, we also propose a novel Constraint-aware Adversarial Decoder (CAD) module, which makes SMART the first generator-based method that could produce adaptive adversarial patterns according to different constraints on attack intensity. Extensive experimental results in various settings demonstrate the effectiveness of the proposed method.

Reversible data hiding aims to embed data into cover digital media in a reversible way. The key issue of image reversible data hiding is to construct the sharply distributed prediction error histogram using advanced pixel prediction. Inspired by the progress of image super-resolution exploiting convolutional neural networks (CNN), CNN predictors can improve the prediction accuracy compared with conventional predictors generally. However, CNN predictors fail to achieve the best prediction accuracy in some cases due to the dependence on training data. To remedy this drawback, the CNN-driven optimal prediction for image reversible data hiding is proposed in this paper. Instead of only utilizing one specific predictor for prediction error histogram construction, the optimal prediction mechanism is designed by incorporating CNN predictors and conventional predictors. Extensive experiments demonstrate the merits of the proposed method in terms of prediction accuracy and marked image quality.

Persistent Fault Attack (PFA) is a recently proposed Fault Attack (FA) method in CHES 2018. It is able to recover full AES secret key in the Single-Byte-Fault scenario. It is demonstrated that classical FA countermeasures, such as Dual Modular Redundancy (DMR) and mask protection, are unable to thwart PFA. In this paper, we propose a fast-detection and fault-correction algorithm to prevent PFA. We construct a fixed input and output pair to detect faults rapidly. Then we build two extra redundant tables to store the relationship between the adjacent elements in the S-box, by which the algorithm can correct the faulty elements in the S-box. Our experimental results show that our algorithm can effectively prevent PFA in both Single-Byte-Fault and Multiple-Bytes-Faults scenarios. Compared with the classical FA countermeasures, our algorithm has a much better effect against PFA. Further, the time cost of our algorithm is 40% lower than the classical FA countermeasures.

Image-to-image translation tasks which have been widely investigated with generative adversarial networks (GAN) aim to map an image from the source domain to the target domain. The translated image can be inversely mapped to the reconstructed source image. However, existing GAN-based schemes lack the ability to accomplish reversible translation. To remedy this drawback, a nearly reversible image-to-image translation scheme where the reconstructed source image is approximately distortion-free compared with the corresponding source image is proposed in this paper. The proposed scheme jointly considers inter-frame coding and embedding. Firstly, we organize the GAN-generated reconstructed source image and the source image into a pseudo video. Furthermore, the bitstream obtained by inter-frame coding is reversibly embedded in the translated image for nearly lossless source image reconstruction. Extensive experimental results and analysis demonstrate that the proposed scheme can achieve a high level of performance in image quality and security.

Over the past several years, to solve the problem of malicious abuse of facial manipulation technology, face manipulation detection technology has obtained considerable attention and achieved remarkable progress. However, most existing methods have very impoverished generalization ability and robustness. In this paper, we propose a novel method for face manipulation detection, which can improve the generalization ability and ro-bustness by bag-of-feature. Specifically, we extend Transformers using bag-of-feature approach to encode inter-patch relation-ships, allowing it to learn forgery features without any additional mask supervision. Extensive experiments demonstrate that our method can outperform competing for state-of-the-art methods on FaceForensics++, Celeb-DF and DeeperForensics-l.0 datasets.

Human activity monitoring (HAM) in the home environment has become increasingly important due to its broad applications including elder care, and well-being management. Recently, some state-of-the-art WiFi-based HAM systems have been proposed due to its properties of non-intrusive and privacy-friendly. However, their key drawback lies in ignoring the crucial impact of human position on HAM. To solve this problem, we present a two-layer WiFi-based HAM system (WiLay), which combines human activity recognition (HAR) with indoor human location (IHL) to provide more integrated information for HAM. Specifically, in the first layer, WiLay adopts the high-frequency energy (HFE) feature of WiFi signals to detect human moving. Then, in the second layer, different processing methods are employed for processing different types of motions accordingly. When the subject activities are static (SAs, the activity without position change), e.g., standing and sitting, WiLay locates the subject before recognizing the specific motion. On the contrary, when the activities are the moving activities (MAs), to reduce the loss of motion information, WiLay employs a comprehensive classifier generated by all different subcarrier classifiers voting, to recognize these MAs accurately. Extensive experimental results show that WiLay has high accuracy with a 99.9% SA/MA detection accuracy rate in the first layer, and a 99.7% location accuracy rate with 98.1% recognition performance for SAs and 90.2% recognition performance for MAs in the second layer.

Recently, truth discovery in crowdsensing systems has received considerable attention with its appealing features for extracting truthful information from multiple unreliable data sources. However, it also poses new challenges to the issues of privacy and security. On the one hand, workers' sensed data can be used to infer their privacy. On the other hand, workers may be selfish and lazy, especially in the Internet-of-Things environment, devices are usually resource constrained, so they may dishonestly execute the costly sensing task so as to reduce resource consumption, or even break the protocol to obtain illegal rewards. Although some privacy-preserving truth discovery schemes have been proposed, they still cannot achieve strong privacy protection while keeping efficiency on the worker side, and still has no efficient incentive mechanism to persuade workers to participate in the system operations. In this article, we propose an incentive-based privacy-preserving truth discovery framework, named InPPTD. By adopting the Paillier homomorphic cryptosystem and two noncolluding servers, InPPTD not only effectively protects workers' sensed data information but also preserves the privacy of these workers' weight information. Meanwhile, a weight-based incentive mechanism is introduced in InPPTD to reduce the number of lazy workers. Security and performance analysis shows that InPPTD can guarantee stronger security features, while also ensure efficiency in terms of computation and communication overhead.

Image style transfer is to re-render the content of one image with the style of another. Most existing methods couple content and style information in their network structures and hyper-parameters, and learn it as a black-box. For better understanding, this paper aims to provide a new explicit decoupled perspective. Specifically, we propose StyleBank, which is composed of multiple convolution filter banks and each filter bank explicitly represents one style. To transfer an image to a specific style, the corresponding filter bank is operated on the intermediate feature produced by a single auto-encoder. The StyleBank and the auto-encoder are jointly learnt in such a way that the auto-encoder does not encode any style information. This explicit representation also enables us to conduct incremental learning to add a new style and fuse styles at not only the image level, but also the region level. Our method is the first style transfer network that links back to traditional texton mapping methods, and provides new understanding on neural style transfer. We further apply this general filterbank learning idea to two different multi-parameter image processing tasks: edge-aware image smoothing and denoising. Experiments demonstrate that it can achieve comparable results to its single parameter setting counterparts.

Many different deep networks have been used to approximate, accelerate or improve traditional image operators. Among these traditional operators, many contain parameters which need to be tweaked to obtain the satisfactory results, which we refer to as "parameterized image operators". However, most existing deep networks trained for these operators are only designed for one specific parameter configuration, which does not meet the needs of real scenarios that usually require flexible parameters settings. To overcome this limitation, we propose a new decoupled learning algorithm to learn from the operator parameters to dynamically adjust the weights of a deep network for image operators, denoted as the base network. The learned algorithm is formed as another network, namely the weight learning network, which can be end-to-end jointly trained with the base network. Experiments demonstrate that the proposed framework can be successfully applied to many traditional parameterized image operators. To accelerate the parameter tuning for practical scenarios, the proposed framework can be further extended to dynamically change the weights of only one single layer of the base network while sharing most computation cost. We demonstrate that this cheap parameter-tuning extension of the proposed decoupled learning framework even outperforms the state-of-the-art alternative approaches.

Reversible Data Hiding (RDH) in JPEG images is valuable for many applications, such as archive management and image authentication. Recently, there emerged a lot of related works for JPEG RDH, however, the current methods utilize the histogram-shifting-based framework in a constant distortion metric, which does not consider the property of DCT coefficients whose modification distortion is miscellaneous with respect to DCT frequency. In this paper, we propose a novel JPEG RDH scheme under multi-distortion metric. At first, the modification distortion is defined as the impact in the spatial domain caused by modifying DCT coefficients. Different from previous works, here we select coefficients with values ±1, ±2 as cover, which will not cause extra bitrate-expansion and decrease the shift distortion. By minimizing both the modification distortion and shift distortion, a heuristic block selecting strategy is proposed. With the selected coefficients and the corresponding distortion, we use recursive histogram modification under the inconsistent-distortion metric for message embedding. The experimental results show that the proposed method can effectively improve visual quality as well as bring small bitstream expansion. Additionally, the undetectability of the proposed method outperforms those of the existing methods.

Traditional watermarking algorithms have been extensively studied. As an important type of watermarking schemes, template-based approaches maintain a very high embedding rate. In such scheme, the message is often represented by some dedicatedly designed templates, and then the message embedding process is carried out by additive operation with the templates and the host image. To resist potential distortions, these templates often need to contain some special statistical features so that they can be successfully recovered at the extracting side. But in existing methods, most of these features are handcrafted and too simple, thus making them not robust enough to resist serious distortions unless very strong and obvious templates are used. Inspired by the powerful feature learning capacity of deep neural network, we propose the first deep template-based watermarking algorithm in this paper. Specifically, at the embedding side, we first design two new templates for message embedding and locating, which is achieved by leveraging the special properties of human visual system, i.e., insensitivity to specific chrominance components, the proximity principle and the oblique effect. At the extracting side, we propose a novel two-stage deep neural network, which consists of an auxiliary enhancing sub-network and a classification sub-network. Thanks to the power of deep neural networks, our method achieves both digital editing resilience and camera shooting resilience based on typical application scenarios. Through extensive experiments, we demonstrate that the proposed method can achieve much better robustness than existing methods while guaranteeing the original visual quality.

As an efficient watermark attack method, geometric distortions destroy the synchronization between the watermark encoder and decoder. Local geometric distortion is a considerable challenge in the watermarking field. Although many geometric distortion resilient watermarking schemes have been proposed, few perform well against local geometric distortions, such as random bending attacks (RBAs). To address this problem, this paper proposes a novel watermark synchronization process and a corresponding watermarking scheme. In our scheme, the watermark bits are represented by random patterns. The message is encoded to obtain a watermark unit, and the watermark unit is flipped to generate a symmetrical watermark. Then, the symmetrical watermark is additively embedded into the spatial domain of the host image. In watermark extraction, we first obtain the theoretical mean-square error minimized estimation of the watermark. Then, an autoconvolution function is applied to this estimation to detect the symmetry and obtain a watermark unit map. According to this map, the watermark can be accurately synchronized, and then extraction can be performed. Experimental results demonstrate the excellent robustness of the proposed watermarking scheme to local geometric distortions, global geometric distortions, common image processing operations, and some kinds of combined attacks.

Many of the recent methods for semi-supervised video object segmentation are still far from being applicable for real time applications due to their slow inference speed. Therefore, we explore a propagation based segmentation method in compressed domain to accelerate inference speed in this paper. In particular, we only extract the features of I-frames by traditional deep convolutional neural network and produce the features of P-frames through information flow propagation. In the process of feature propagation, we propose two effective components to enhance the representation ability of simply warped features in terms of appearance and location. Specifically, we propose a residual supplement module to supplement appearance information which is lost in direct warping and a spatial attention module that can mine extra spatial saliency to provide the location information of the specified object. Besides, we propose a metric based decoder module which consists of a feature match module and a multi-level refinement module to transform information from semantic representation to shape segmentation mask. Extensive experiments on several video datasets demonstrate that the proposed method can achieve comparable accuracy while much faster inference speed when compared to the state-of-the-art algorithms.

Recently, Li et al. proposed an effective non-additive distortion model for JPEG steganography by preserving Block Boundary Continuity (BBC) in the spatial domain. However, the method based on BBC only explored how the modifications of DCT coefficient pairs at the same mode in adjacent blocks will impact on BBC. In this paper, we propose a method to enhance the BBC, called BBC++, by considering the mutual impact on the BBC from all DCT coefficients in adjacent blocks. To do that, we design updating strategies for both covers and costs in multi-round embedding processions. During the embedding, the cover is updated to repair the BBC after embedding and the corresponding costs are updated to keep the BBC from being destroyed in the next embedding. Experimental results show that the BBC++ can better maintain BBC and outperform previous non-additive distortion steganography when resisting modern JPEG steganalyzers.

Recent advances show that a reasonable non-additive cost function can significantly improve the security level of additive cost based steganography. So far, there is only one principle, called block boundary continuity (BBC), that has been proposed to define the non-additive cost function for JPEG steganography, and it aims at synchronizing the modification direction of inter-block boundaries in the spatial domain. In this article, we found that JPEG steganography usually introduces more and larger modifications on the boundary than on the inside of each intra-block in the spatial domain, which is another important factor affecting security. Therefore, we present a new principle, called block boundary maintenance (BBM), to minimize the modifications on the spatial block boundaries. In theory, we deduce the BBM principle on how to modify a pair of DCT coefficients of the intra-block to reduce the modifications on the spatial block boundary. According to the BBM principle, we design a new strategy to define non-additive cost functions for JPEG steganography by exploiting the coefficient correlation of the intra-block in the DCT domain. The experimental results show that the BBM-based strategy can minimize modifications on the spatial block boundaries and thus achieve a high-security level when resisting modern JPEG steganalysis. Furthermore, the two principles of BBC and BBM can be fused to further improve the empirical security.

The standard tensor voting technique shows its versatility in tasks such as object recognition and semantic segmentation by recognizing feature points and sharp edges that can segment a model into several patches. We propose a neighborhood-level representation-guided tensor voting model for 3D mesh steganalysis. Because existing steganalytic methods do not analyze correlations among neighborhood faces, they are not very effective at discriminating stego meshes from cover meshes. In this paper, we propose to utilize a tensor voting model to reveal the artifacts caused by embedding data. In the proposed steganalytic scheme, the normal voting tensor (NVT) operation is performed on original mesh faces and smoothed mesh faces separately. Then, the absolute values of the differences between the eigenvalues of the two tensors (from the original face and the smoothed face) are regarded as features that capture intricate relationships among the vertices. Subsequently, the extracted features are processed with a nonlinear mapping to boost the feature effectiveness. The experimental results show that the proposed feature sets prevail over state-of-the-art feature sets including LFS64 and ELFS124 under various steganographic schemes.

In this paper, we propose an online multi-object tracking (MOT) approach that integrates data association and single object tracking (SOT) with a unified convolutional network (ConvNet), named DASOTNet. The intuition behind integrating data association and SOT is that they can complement each other. Following Siamese network architecture, DASOTNet consists of the shared feature ConvNet, the data association branch and the SOT branch. Data association is treated as a special re-identification task and solved by learning discriminative features for different targets in the data association branch. To handle the problem that the computational cost of SOT grows intolerably as the number of tracked objects increases, we propose an efficient two-stage tracking method in the SOT branch, which utilizes the merits of correlation features and can simultaneously track all the existing targets within one forward propagation. With feature sharing and the interaction between them, data association branch and the SOT branch learn to better complement each other. Using a multi-task objective, the whole network can be trained end-to-end. Compared with state-of-the-art online MOT methods, our method is much faster while maintaining a comparable performance.

Deep learning has achieved tremendous success in numerous industrial applications. As training a good model often needs massive high-quality data and computation resources, the learned models often have significant business values. However, these valuable deep models are exposed to a huge risk of infringements. For example, if the attacker has the full information of one target model including the network structure and weights, the model can be easily finetuned on new datasets. Even if the attacker can only access the output of the target model, he/she can still train another similar surrogate model by generating a large scale of input-output training pairs. How to protect the intellectual property of deep models is a very important but seriously under-researched problem. There are a few recent attempts at classification network protection only.In this paper, we propose the first model watermarking framework for protecting image processing models. To achieve this goal, we leverage the spatial invisible watermarking mechanism. Specifically, given a black-box target model, a unified and invisible watermark is hidden into its outputs, which can be regarded as a special task-agnostic barrier. In this way, when the attacker trains one surrogate model by using the input-output pairs of the target model, the hidden watermark will be learned and extracted afterward. To enable watermarks from binary bits to high-resolution images, both traditional and deep spatial invisible watermarking mechanism are considered. Experiments demonstrate the robustness of the proposed watermarking mechanism, which can resist surrogate models learned with different network structures and objective functions. Besides deep models, the proposed method is also easy to be extended to protect data and traditional image processing algorithms.

In this paper, we look into the problem of 3D adversary attack, and propose to leverage the internal properties of the point clouds and the adversarial examples to design a new self-robust deep neural network (DNN) based 3D recognition systems. As a matter of fact, on one hand, point clouds are highly structured. Hence for each local part of clean point clouds, it is possible to learn what is it (``part of a bottle") and its relative position (``upper part of a bottle") to the global object center. On the other hand, with the visual quality constraint, 3D adversarial samples often only produce small local perturbations, thus they will roughly keep the original global center but may cause incorrect local relative position estimation. Motivated by these two properties, we use relative position (dubbed as ``gather-vector") as the adversarial indicator and propose a new robust gather module. Equipped with this module, we further propose a new self-robust 3D point recognition network. Through extensive experiments, we demonstrate that the proposed method can improve the robustness of the target attack under the white-box setting significantly. For I-FGSM based attack, our method reduces the attack success rate from 94.37 \% to 75.69 \%. For C\&W based attack, our method reduces the attack success rate more than 40.00 \%. Moreover, our method is complementary to other types of defense methods to achieve better defense results.

Deep Neural Networks are vulnerable to adversarial samples, which can fool classifiers by adding small perturbations onto the original image. Since the pioneering optimization-based adversarial attack method, many following methods have been proposed in the past several years. However most of these methods add perturbations in a "pixel-wise" and "global" way. Firstly, because of the contradiction between the local smoothness of natural images and the noisy property of these adversarial perturbations, this "pixel-wise" way makes these methods not robust to image processing based defense methods and steganalysis based detection methods. Secondly, we find adding perturbations to the background is less useful than to the salient object, thus the "global" way is also not optimal. Based on these two considerations, we propose the first robust superpixel-guided attentional adversarial attack method. Specifically, the adversarial perturbations are only added to the salient regions and guaranteed to be same within each superpixel. Through extensive experiments, we demonstrate our method can preserve the attack ability even in this highly constrained modification space. More importantly, compared to existing methods, it is significantly more robust to image processing based defense and steganalysis based detection.

Semi-supervised learning (SSL) has been extensively studied to improve the generalization ability of deep neural networks for visual recognition. To involve the unlabelled data, most existing SSL methods are based on common density-based cluster assumption: samples lying in the same high-density region are likely to belong to the same class, including the methods performing consistency regularization or generating pseudo-labels for the unlabelled images. Despite their impressive performance, we argue three limitations exist: 1) Though the density information is demonstrated to be an important clue, they all use it in an implicit way and have not exploited it in depth. 2) For feature learning, they often learn the feature embedding based on the single data sample and ignore the neighborhood information. 3) For label-propagation based pseudo-label generation, it is often done offline and difficult to be end-to-end trained with feature learning. Motivated by these limitations, this paper proposes to solve the SSL problem by building a novel density-aware graph, based on which the neighborhood information can be easily leveraged and the feature learning and label propagation can also be trained in an end-to-end way. Specifically, we first propose a new Density-aware Neighborhood Aggregation(DNA) module to learn more discriminative features by incorporating the neighborhood information in a density-aware manner. Then a novel Density-ascending Path based Label Propagation(DPLP) module is proposed to generate the pseudo-labels for unlabeled samples more efficiently according to the feature distribution characterized by density. Finally, the DNA module and DPLP module evolve and improve each other end-to-end. Extensive experiments demonstrate the effectiveness of the newly proposed density-aware graph based SSL framework and our approach can outperform current state-of-the-art methods by a large margin.

Cross-modality person re-identification (cm-ReID) is a challenging but key technology for intelligent video analysis. Existing works mainly focus on learning modality-shared representation by embedding different modalities into a same feature space, lowering the upper bound of feature distinctiveness. In this paper, we tackle the above limitation by proposing a novel cross-modality shared-specific feature transfer algorithm (termed cm-SSFT) to explore the potential of both the modality-shared information and the modality-specific characteristics to boost the reidentification performance. We model the affinities of different modality samples according to the shared features and then transfer both shared and specific features among and across modalities. We also propose a complementary feature learning strategy including modality adaption, project adversarial learning and reconstruction enhancement to learn discriminative and complementary shared and specific features of each modality, respectively. The entire cmSSFTalgorithm can be trained in an end-to-end manner. We conducted comprehensive experiments to validate the superiority ofthe overall algorithm and the effectiveness ofeach component. The proposed algorithm significantly outperforms state-of-the-arts by 22.5% and 19.3% mAP on the two mainstream benchmark datasets SYSU-MM01 and RegDB, respectively.

Deep neural networks have made tremendous progress in 3D point-cloud recognition. Recent works have shown that these 3D recognition networks are also vulnerable to adversarial samples produced from various attack methods, including optimization-based 3D Carlini-Wagner attack, gradient-based iterative fast gradient method, and skeleton-detach based point-dropping. However, after a careful analysis, these methods are either extremely slow because of the optimization/iterative scheme, or not flexible to support targeted attack of a specific category. To overcome these shortcomings, this paper proposes a novel label guided adversarial network (LG-GAN) for real-time flexible targeted point cloud attack. To the best of our knowledge, this is the first generation based 3D point cloud attack method. By feeding the original point clouds and target attack label into LG-GAN, it can learn how to deform the point clouds to mislead the recognition network into the specific label only with a single forward pass. In detail, LG-GAN first leverages one multi-branch adversarial network to extract hierarchical features of the input point clouds, then incorporates the specified label information into multiple intermediate features using the label encoder. Finally, the encoded features will be fed into the coordinate reconstruction decoder to generate the target adversarial sample. By evaluating different point-cloud recognition models (e.g., PointNet, PointNet++ and DGCNN), we demonstrate that the proposed LG-GAN can support flexible targeted attack on the fly while guaranteeing good attack performance and higher efficiency simultaneously.

Recent work has demonstrated that neural networks are vulnerable to adversarial examples. To escape from the predicament, many works try to harden the model in various ways, in which adversarial training is an effective way which learns robust feature representation so as to resist adversarial attacks. Meanwhile, the self-supervised learning aims to learn robust and semantic embedding from data itself. With these views, we introduce self-supervised learning to against adversarial examples in this paper. Specifically, the self-supervised representation coupled with k-Nearest Neighbour is proposed for classification. To further strengthen the defense ability, self-supervised adversarial training is proposed, which maximizes the mutual information between the representations of original examples and the corresponding adversarial examples. Experimental results show that the self-supervised representation outperforms its supervised version in respect of robustness and self-supervised adversarial training can further improve the defense ability efficiently.

Video object detection is a challenging problem in computer vision. In this paper, we propose a novel spatial-temporal feature aggregation network to deal with this issue. Specifically, we present a novel instance-level feature aggregation module as complementary to traditional pixel-level feature aggregation, in which we build a new movement estimation module to learn instance movements across frames. Then the Graph Convolutional Networks (GCNs) is applied to obtain temporal relation among instances over frames to implement instance-level feature aggregation. At last, we combine pixel-level and instance-level features by learnable soft weights to make use of their complementary information. Our framework is simple to implement and enables end-to-end training, which achieves state-of-art performance on the ImageNet VID dataset by extensive experiments.

The popular tracking-by-detection paradigm for multi-object tracking (MOT) focuses on solving data association problem, of which a robust similarity model lies in the heart. Most previous works make effort to improve feature representation for individual object while leaving the relations among objects less explored, which may be problematic in some complex scenarios. In this paper, we focus on leveraging the relations among objects to improve robustness of the similarity model. To this end, we propose a novel graph representation that takes both the feature of individual object and the relations among objects into consideration. Besides, a graph matching module is specially designed for the proposed graph representation to alleviate the impact of unreliable relations. With the help of the graph representation and the graph matching module, the proposed graph similarity model, named GSM, is more robust to the occlusion and the targets sharing similar appearance. We conduct extensive experiments on challenging MOT benchmarks and the experimental results demonstrate the effectiveness of the proposed method.

Human activity recognition (HAR) has become increasingly essential due to its potential to support a broad array of applications, e.g., elder care, and VR games. Recently, some pioneer WiFi-based HAR systems have been proposed due to its privacy-friendly and device-free characteristics. However, their crucial limitation lies in ignoring the inevitable impact of co-channel interference (CCI), which degrades the performance of these HAR systems significantly. To address this challenge, we propose PhaseAnti, a novel HAR system to exploit the CCI- independent phase component, NLPEV (Nonlinear Phase Error Variation), of Channel State Information (CSI) to cope with the impact of CCI. We provide a rigorous analysis of NLPEV data with respect to its stability and otherness. Validated by our experiments, this phase component across subcarriers is invariant to various CCI scenarios, while different for distinct motions. Based on the analysis, we use NLPEV data to perform HAR in CCI scenarios. Extensive experiments demonstrate that PhaseAnti can reliably recognize activity in various CCI scenarios. Specifically, PhaseAnti achieves a 95% recognition accuracy rate (RAR) on average, which improves up to 16% RAR in the presence of CCI. Moreover, the recognition speed is 9× faster than the state-of-the-art solution.

Deep convolutional neural networks have made outstanding contributions in many fields such as computer vision in the past few years and many researchers published well-trained network for downloading. But recent studies have shown serious concerns about integrity due to model-reuse attacks and backdoor attacks. In order to protect these open-source networks, many algorithms have been proposed such as watermarking. However, these existing algorithms modify the contents of the network permanently and are not suitable for integrity authentication. In this paper, we propose a reversible watermarking algorithm for integrity authentication. Specifically, we present the reversible watermarking problem of deep convolutional neural networks and utilize the pruning theory of model compression technology to construct a host sequence used for embedding watermarking information by histogram shift. As shown in the experiments, the influence of embedding reversible watermarking on the classification performance is less than ±0.5% and the parameters of the model can be fully recovered after extracting the watermarking. At the same time, the integrity of the model can be verified by applying the reversible watermarking: if the model is modified illegally, the authentication information generated by original model will be absolutely different from the extracted watermarking information.

As a newly added in-loop filtering technique in High Efficiency Video Coding (HEVC), sample adaptive offset (SAO) can be utilized to embed messages for video steganography. This paper presents a novel SAO-based HEVC video steganographic scheme. The main principle is to design a suitable distortion function which expresses the embedding impacts on offsets based on minimizing embedding distortion. Two factors including the sample rate-distortion cost fluctuation and the sample statistical characteristic are considered in embedding distortion definition. Adaptive message embedding is implemented using syndrome-trellis codes (STC). Experimental results demonstrate the merits of the proposed scheme in terms of undetectability and video coding performance.

Modern deep neural networks(DNNs) are vulnerable to adversarial samples. Sparse adversarial samples are a special branch of adversarial samples that can fool the target model by only perturbing a few pixels. The existence of the sparse adversarial attack points out that DNNs are much more vulnerable than people believed, which is also a new aspect for analyzing DNNs. However, current sparse adversarial attack methods still have some shortcomings on both sparsity and invisibility. In this paper, we propose a novel two-stage distortion-aware greedy-based method dubbed as ''GreedyFool". Specifically, it first selects the most effective candidate positions to modify by considering both the gradient(for adversary) and the distortion map(for invisibility), then drops some less important points in the reduce stage. Experiments demonstrate that compared with the start-of-the-art method, we only need to modify 3 times fewer pixels under the same sparse perturbation setting. For target attack, the success rate of our method is 9.96% higher than the start-of-the-art method under the same pixel budget.

Despite tremendous success in many application scenarios, deep learning faces serious intellectual property (IP) infringement threats. Considering the cost of designing and training a good model, infringements will significantly infringe the interests of the original model owner. Recently, many impressive works have emerged for deep model IP protection. However, they either are vulnerable to ambiguity attacks, or require changes in the target network structure by replacing its original normalization layers and hence cause significant performance drops. To this end, we propose a new passport-aware normalization formulation, which is generally applicable to most existing normalization layers and only needs to add another passport-aware branch for IP protection. This new branch is jointly trained with the target model but discarded in the inference stage. Therefore it causes no structure change in the target model. Only when the model IP is suspected to be stolen by someone, the private passport-aware branch is added back for ownership verification. Through extensive experiments, we verify its effectiveness in both image and 3D point recognition models. It is demonstrated to be robust not only to common attack techniques like fine-tuning and model compression, but also to ambiguity attacks. By further combining it with trigger-set based methods, both black-box and white-box verification can be achieved for enhanced security of deep learning models deployed in real systems.

With the advances in wireless communication and Internet of things, wireless body area networks (WBANs) have attracted more and more attention because of the potential in improving the quality of health care services. With the help of WBANs, the user can access the patient's life‐critical data generated by miniaturised medical sensors, and remote health care monitoring services are provided. Since the open nature of wireless channel and sensitivity of transmitted information, the security and privacy of such personal data are becoming important issues that must be dealt with. In the past few years, a large number of authentication schemes had been proposed to solve these issues. However, most of the existing schemes are not secure enough. As a step toward this direction, in this study, the authors present a privacy‐preserving authentication scheme with adaptive resilience of desynchronisation attacks for WBANs, in which lightweight crypto‐modules are adopted to pursue the best efficiency. The proposed scheme adopts the pseudonym identity technique to provide user anonymity, and one‐way hash chain technique and serial number method are employed to ensure forward secrecy and resist desynchronisation attack, respectively. Analysis and comparison results demonstrate that the proposed scheme achieves a delicate balance between security and efficiency.

This article describes a fast Reed-Solomon encoding algorithm with four and seven parity symbols in between. First, we show that the syndrome of Reed-Solomon codes can be computed via the Reed-Muller transform. Based on this result, the fast encoding algorithm is then derived. Analysis shows that the proposed approach asymptotically requires 3 XORs per data bit, representing an improvement over previous algorithms. The simulation demonstrates that the performance of the proposed approach improves with the increase of code length and is superior to other methods. In particular, when the parity number is 5, the proposed approach is about two times faster than other cutting-edge methods.

Steganography is an information hiding technique for covert communication. So far Syndrome-Trellis Codes (STC), a convolutional codes-based method, is the only near-optimal coding method, i.e., it can approach the rate-distortion bound of content-adaptive steganography in practice. However, as a secure communication application, steganography needs the diversity of coding methods. This paper proposes another and a better near-optimal steganographic coding method based on polar codes, using Successive Cancellation List (SCL) decoding algorithm to minimize additive distortion in steganography. Considering a steganographic channel as a binary symmetric channel, the proposed Steganographic Polar Codes (SPC) chooses parity-check matrix by setting embedding payload as the initial value of Arikan's heuristic and computes decoding channel metric from the optimal modification probability of minimal distortion model. To overcome the inherent defect of polar codes only suiting for code length of a power of 2, we introduce three strategies to generalize SPC for arbitrary length. Experimental results validate the versatility of SPC to minimize arbitrary distortion. When compared with STC, the overall coding performance of SPC is more superior with low embedding complexity. This work verifies the availability of polar codes for the practical construction of steganographic codes and provides a methodology for designing better steganographic codes based on any advance of polar coding/decoding.

Golomb coding is a type of entropy encoding scheme for geometric distributions. It consists of two parts, and both parts are coded with variable-length coding, which requires a higher computational effort than fixed-length coding schemes. To solve this issue, the first part of this article presents a variant of Golomb coding that uses fixed-length coding to code the first part. The simulations show that the proposed coding scheme has a higher throughput than Golomb coding, due to the reduction of arithmetic complexity. In the second part, we discuss the n-ary versions of Golomb coding and the proposed coding scheme.

Steganography is the art of covert communication, which aims to hide the secret messages into cover medium while achieving high undetectability. To this end, the framework of minimal distortion embedding is widely adopted for adaptive steganography, where a well-designed distortion function is significant. In this paper, inspired by the phenomenon that the modification of audio samples with the low amplitude will be easily detected, a novel distortion is presented for audio steganography. Taking the fragility of the low amplitude audio samples into account, the proposed distortion is inversely proportional to the amplitude. Furthermore, in order to resist the strong steganalysis, the derivative filter is utilized for acquiring the residual of audio, which plays an important role in distortion definition. The experimental results show that the proposed distortion outperforms the state-of-the-art methods defending strong steganalytic methods. To take a step forward, considering the mutual impact caused by embedding modification, the non-additive extensions of the proposed methods are put forward. The extending experiments show that in most cases, the proposed non-additive extensions can achieve higher level of security than the original methods.

This paper designs a novel underpainting based camera shooting resilient (CSR) document watermarking algorithm for dealing with the leak source tracking problem. By applying such algorithm, we can extract the authentication watermark information from the candid photographs. The watermarked underpainting contains three significant properties. 1) Inconspicuousness. The watermarked underpainting is inconspicuous and it will not easily be maliciously attacked. 2) Robustness. We propose DCT-based watermark embedding algorithm and distortion compensation based extracting algorithm, which make the watermark robust to camera shooting process. 3) Autocorrelation. We design the flip-based method to arrange the watermarked underpainting. So that a complete watermark region can be accurately located even if part of the document is recorded. Compared with previous watermarking algorithms, the proposed scheme guaranteed content independent embedding as well as the robustness to the camera shooting process. Besides, the proposed scheme satisfies the accuracy of extraction even when the captured document is incomplete.

Previous studies have exhibited that incorporating side-information, e.g., a high-quality precover image, can significantly improve steganographic security for JPEG images. This motivates us to estimate the side-information for traditional steganographic scenario in which only a JPEG image is available. It is expected to achieve high-level security by utilizing the estimated side-information similar to side-informed steganography, even though the estimated side-information is not perfectly precise. In this paper, a general framework of side-information estimated (SIE) JPEG steganography is proposed, under which the core problems are how to better estimate the precover and modulate the distortion function correspondingly. To address the two problems, we test several denoising filters and a deblocking filter to obtain the estimated precover, and we introduce two implementation models for modulating the costs. We finally recommend the combination of the deblocking filter and the modulation model using the polarity of the estimated rounding error. The experimental results show that the proposed method dramatically improves the existing additive distortions for images of an arbitrary quality factor and outperforms the state-of-the-art methods based on estimating side-information when resisting modern steganalysis.

Recently, the most effective steganographic schemes for JPEG images are based on minimal distortion model with Syndrome-Trellis Codes (STCs) as the coding method. However, the execution time of STCs will be severely long for message embedding to the cover object of large size, which cannot meet the demand for real-time communication in a real-world application. According to the time complexity O(2hn), it is suggested in the STCs to accelerate the embedding process by decreasing the constraint height h. However, smaller h corresponds to lower steganographic security. In this paper, we investigate the possibility of shortening the cover (reducing the length n) for speeding up the execution of STCs without weakening the steganographic security. After introducing some properties of cover selection with proofs, we propose several algorithms designed for JPEG images to construct a preferable shortened cover containing DCT coefficients of smaller costs as much as possible. The experimental results display the superiority of the proposed algorithm on the speed profit and the security when compared with the method of decreasing h. With confidence, a JPEG image of arbitrary quality factor can be safely shortened to 1/4 of the original, and correspondingly the execution of STCs can be four times faster.

Smart grid adopts two-way communication and rich functionalities to gain a positive impact on the sustainability and efficiency of power usage, but on the other hand, also poses serious challenges to customers' privacy. Existing solutions in smart grid usually use cryptographic tools, such as homomorphic encryption, to protect individual privacy, which, however, can only support limited and simple functionalities. Moreover, the resource-constrained smart meters need to perform heavy asymmetric cryptography in these solutions, and thus unnecessarily increases load on smart grid. In this paper, we present a practical and secure SGX-enabled smart grid system, named SecGrid. Our system leverages trusted hardware SGX to ensure that grid utilities can efficiently execute rich functionalities on customers' private data, while guaranteeing their privacy. With our well-devised security protocols in SecGrid, only the smart meters need to perform AES encryption. To validate the superiority of our design, we conduct security analysis and experimentation. Security analysis shows that SecGrid can thwart various attacks from malicious adversaries, and the experimental results show that SecGrid is much faster than the existing privacy-preserving schemes in smart grid.

Despite the strong presence of color images for communication, scholars have mainly devoted their attention to research on steganography for grayscale images. In contrast to grayscale images, color images have three interrelated color channels, and the relationships among the three channels have a strong impact on the steganography security. In this paper, we present a steganographic scheme for spatial color images by exploiting the correlations and differences between the color channels. We find that the G channel has a stronger correlation with R and B than the one between R and B, and thus, synchronizing the modification directions of the R and B channels with those from the G channel will have better resistance to detection. In addition, the payload capacity and the distribution of complex regions between channels are different. Based on these findings, we design a new strategy for defining non-additive costs for color image steganography, called G-channel-related Inter-channel Non-Additive (GINA) strategy. The GINA strategy can make the modification directions of the R and B channels consistent with those of the G channel and can adaptively distribute the embedding capacity between the three channels. Specifically, this strategy will not violate the Complexity Prior rule. The experimental results show that the proposed GINA strategy can significantly improve the performance in terms of resisting color image steganalysis compared with previous methods.

Traditional image processing operators often provide some control parameters to tweak the final results. Recently, different convolutional neural networks have been used to approximate or improve these operators. However, in those methods, one single model can only handle one operator of a specific parameter value and does not support parameter tuning. In this paper, we propose a new plugin module, “Adaptive Filterbank Pyramid”, which can be inserted into a backbone network to support multiple operators and continuous parameter tuning. Our module explicitly represents one operator with one filterbank pyramid. To generate the results of a specific operator, the corresponding filterbank pyramid is convolved with the intermediate feature pyramid produced by the backbone network. The weights of the filterbank pyramid are directly regressed by another sub-network, which is jointly trained with the backbone network and adapted to the input parameter, thus enabling continuous parameter tuning. We applied the proposed module for a large variety of image processing tasks, including image smoothing, image denoising, image deblocking, image enhancement and neural style transfer. Experiments show that our method is generalized to different types of image processing tasks and different backbone network structures. Compared to the single-operator-single-parameter baseline, our method can produce comparable results but is significantly more efficient in both training and testing.

In recent years, the field of object detection has made significant progress. The success of most of the state-of-the-art object detectors is derived from the use of feature pyramid and the carefully designed anchor boxes. However, the current methods of constructing feature pyramid usually blindly integrate multi-scale representations on each feature hierarchy. Furthermore, these detectors also suffer from some drawbacks brought by the hand-designed anchors. To mitigate the adverse effects caused thereby, we introduce a one-stage object detector, named as the semi-anchor-free network with enhanced feature pyramid (SAFNet). Specifically, to better construct feature pyramid, we propose a novel enhanced feature pyramid generation paradigm, which mainly consists of two modules, i.e., adaptive feature fusion module (AFFM) and self-enhanced module (SEM). The paradigm adaptively integrates multi-scale representations in a non-linear method meanwhile suppress the redundant semantic information for each pyramid level, such that a clean and enhanced feature pyramid could be obtained. In addition, an adaptive anchor generator (AAG) is designed to yield fewer but more suitable anchor boxes for each input image. Benefiting from the enhanced feature pyramid, AAG is capable of generating more accurate anchor boxes by introducing few priors. Thus, AAG has the ability to alleviate the drawbacks caused by the preset anchor hyper-parameters and helps to decrease the computation cost. Extensive experiments demonstrate the effectiveness of our approach. Profited from the proposed modules, SAFNet significantly boosts the detection performance, i.e., achieving 2 points and 2.1 points higher Average Precision (AP) than RetinaNet (our baseline) on PASCAL VOC and MS COCO respectively. Codes will be publicly available soon.

Side channel steganalysis refers to detecting a steganographer in social websites via behavior analysis. In this paper, we first design a side channel steganalysis based on the correlation between image sequences of social users, which aims to find out the behaviorally anomalous steganographer. According to the experimental results of side channel steganalysis, it is intuitively secure for the steganographer to act identically to normal social users since she can avoid being detected by side channel steganalysis. However, when faced with various detection methods, is it still secure to behave similar to a normal user? To comprehensively consider the detection means and further explore the secure behavior region of the steganographer, we design a complementary attack of side channel steganalysis. Specifically, we take the correlation of contents of images as side information and take the images with similar content as references to calibrate steganalysis features, which helps improve traditional steganalysis. The proposed side channel steganalysis and its complementary attack efficiently detect steganographers from two different aspects. When the average rank of the steganographer is used to measure the performance, side channel steganalysis can rank the steganographer within the top ten in 100 actors, and the complementary attack can raise the average rank of the steganographer by three places compared with the previous method. From the perspective of the steganographer on social networks, it can help her behave in a more secure region, where her behavior should neither deviate from that of normal users nor be too similar to that of normal users.

Despite the recent success of face image generation with GANs, conditional hair editing remains challenging due to the under-explored complexity of its geometry and appearance. In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation. To provide user control over every major hair visual factor, we explicitly disentangle hair into four orthogonal attributes, including shape, structure, appearance, and background. For each of them, we design a corresponding condition module to represent, process, and convert user inputs, and modulate the image generation pipeline in ways that respect the natures of different visual attributes. All these condition modules are integrated with the backbone generator to form the final end-to-end network, which allows fully-conditioned hair generation from multiple user inputs. Upon it, we also build an interactive portrait hair editing system that enables straightforward manipulation of hair by projecting intuitive and high-level user inputs such as painted masks, guiding strokes, or reference photos to well-defined condition representations. Through extensive experiments and evaluations, we demonstrate the superiority of our method regarding both result quality and user controllability.

The new paradigm of outsourcing data to the cloud is a double-edged sword. On the one hand, it frees data owners from the technical management, and is easier for data owners to share their data with intended users. On the other hand, it poses new challenges on privacy and security protection. To protect data confidentiality against the honest-but-curious cloud service provider, numerous works have been proposed to support fine-grained data access control. However, till now, no schemes can support both fine-grained access control and time-sensitive data publishing. In this paper, by embedding timed-release encryption into Ciphertext-Policy Attribute-based Encryption (CP-ABE), we propose a new time and attribute factors combined access control on time-sensitive data for public cloud storage (named TAFC). Based on the proposed scheme, we further propose an efficient approach to design access policies faced with diverse access requirements for time-sensitive data. Extensive security and performance analysis shows that our proposed scheme is highly efficient and satisfies the security requirements for time-sensitive data storage in public cloud.

Evolution Strategies (ES), a class of black-box optimization algorithms, has recently been demonstrated to be a viable alternative to popular MDP-based RL techniques such as Qlearning and Policy Gradients. ES achieves fairly good performance on challenging reinforcement learning problems and is easier to scale in a distributed setting. However, standard ES algorithms perform one gradient update per data sample, which is not very efficient. In this paper, with the purpose of more efficient using of sampled data, we propose a novel iterative procedure that optimizes a surrogate objective function, enabling to reuse data sample for multiple epochs of updates. We prove monotonic improvement guarantee for such procedure. By making several approximations to the theoretically-justified procedure, we further develop a practical algorithm called Trust Region Evolution Strategies (TRES). Our experiments demonstrate the effectiveness of TRES on a range of popular MuJoCo locomotion tasks in the OpenAI Gym, achieving better performance than ES algorithm.

Recently, path norm was proposed as a new capacity measure for neural networks with Rectified Linear Unit (ReLU) activation function, which takes the rescaling-invariant property of ReLU into account. It has been shown that the generalization error bound in terms of the path norm explains the empirical generalization behaviors of the ReLU neural networks better than that of other capacity measures. Moreover, optimization algorithms which take path norm as the regularization term to the loss function, like Path-SGD, have been shown to achieve better generalization performance. However, the path norm counts the values of all paths, and hence the capacity measure based on path norm could be improperly influenced by the dependency among different paths. It is also known that each path of a ReLU network can be represented by a small group of linearly independent basis paths with multiplication and division operation, which indicates that the generalization behavior of the network only depends on only a few basis paths. Motivated by this, we propose a new norm Basis-path Norm based on a group of linearly independent paths to measure the capacity of neural networks more accurately. We establish a generalization error bound based on this basis path norm, and show it explains the generalization behaviors of ReLU networks more accurately than previous capacity measures via extensive experiments. In addition, we develop optimization algorithms which minimize the empirical risk regularized by the basis-path norm. Our experiments on benchmark datasets demonstrate that the proposed regularization method achieves clearly better performance on the test set than the previous regularization approaches.

Deep Neural Networks (DNNs) have recently led to significant improvements in many fields. However, DNNs are vulnerable to adversarial examples which are samples with imperceptible perturbations while dramatically misleading the DNNs. Moreover, adversarial examples can be used to perform an attack on various kinds of DNN based systems, even if the adversary has no access to the underlying model. Many defense methods have been proposed, such as obfuscating gradients of the networks or detecting adversarial examples. However it is proved out that these defense methods are not effective or cannot resist secondary adversarial attacks. In this paper, we point out that steganalysis can be applied to adversarial examples detection, and propose a method to enhance steganalysis features by estimating the probability of modifications caused by adversarial attacks. Experimental results show that the proposed method can accurately detect adversarial examples. Moreover, secondary adversarial attacks are hard to be directly performed to our method because our method is not based on a neural network but based on high-dimensional artificial features and Fisher Linear Discriminant ensemble.

Synthesizing photo-realistic images from text descriptions is a challenging problem. Previous studies have shown remarkable progresses on visual quality of the generated images. In this paper, we consider semantics from the input text descriptions in helping render photo-realistic images. However, diverse linguistic expressions pose challenges in extracting consistent semantics even they depict the same thing. To this end, we propose a novel photo-realistic text-to-image generation model that implicitly disentangles semantics to both fulfill the high-level semantic consistency and low-level semantic diversity. To be specific, we design (1) a Siamese mechanism in the discriminator to learn consistent high-level semantics, and (2) a visual-semantic embedding strategy by semantic-conditioned batch normalization to find diverse low-level semantics. Extensive experiments and ablation studies on CUB and MS-COCO datasets demonstrate the superiority of the proposed method in comparison to state-of-the-art methods.

Dense captioning aims at simultaneously localizing semantic regions and describing these regions-of-interest (ROIs) with short phrases or sentences in natural language. Previous studies have shown remarkable progresses, but they are often vulnerable to the aperture problem that a caption generated by the features inside one ROI lacks contextual coherence with its surrounding context in the input image. In this work, we investigate contextual reasoning based on multi-scale message propagations from the neighboring contents to the target ROIs. To this end, we design a novel end-to-end context and attribute grounded dense captioning framework consisting of 1) a contextual visual mining module and 2) a multi-level attribute grounded description generation module. Knowing that captions often co-occur with the linguistic attributes (such as who, what and where), we also incorporate an auxiliary supervision from hierarchical linguistic attributes to augment the distinctiveness of the learned captions. Extensive experiments and ablation studies on Visual Genome dataset demonstrate the superiority of the proposed model in comparison to state-of-the-art methods.

Modern deep neural networks are often vulnerable to adversarial samples. Based on the first optimization-based attacking method, many following methods are proposed to improve the attacking performance and speed. Recently, generation-based methods have received much attention since they directly use feed-forward networks to generate the adversarial samples, which avoid the time-consuming iterative attacking procedure in optimization-based and gradient-based methods. However, current generation-based methods are only able to attack one specific target (category) within one model, thus making them not applicable to real classification systems that often have hundreds/thousands of categories. In this paper, we propose the first Multi-target Adversarial Network (MAN), which can generate multi-target adversarial samples with a single model. By incorporating the specified category information into the intermediate features, it can attack any category of the target classification model during runtime. Experiments show that the proposed MAN can produce stronger attack results and also have better transferability than previous state-of-the-art methods in both multi-target attack task and single-target attack task. We further use the adversarial samples generated by our MAN to improve the robustness of the classification model. It can also achieve better classification accuracy than other methods when attacked by various methods.

Learning discriminative image feature embeddings is of great importance to visual recognition. To achieve better feature embeddings, most current methods focus on designing different network structures or loss functions, and the estimated feature embeddings are usually only related to the input images. In this paper, we propose Memory-based Neighbourhood Embedding (MNE) to enhance a general CNN feature by considering its neighbourhood. The method aims to solve two critical problems, i.e., how to acquire more relevant neighbours in the network training and how to aggregate the neighbourhood information for a more discriminative embedding. We first augment an episodic memory module into the network, which can provide more relevant neighbours for both training and testing. Then the neighbours are organized in a tree graph with the target instance as the root node. The neighbourhood information is gradually aggregated to the root node in a bottom-up manner, and aggregation weights are supervised by the class relationships between the nodes. We apply MNE on image search and few shot learning tasks. Extensive ablation studies demonstrate the effectiveness of each component, and our method significantly outperforms the state-of-the-art approaches.

Neural networks are vulnerable to adversarial examples, which poses a threat to their application in security sensitive systems. We propose a Denoiser and UPsampler Network (DUP-Net) structure as defenses for 3D adversarial point cloud classification, where the two modules reconstruct surface smoothness by dropping or adding points. In this paper, statistical outlier removal (SOR) and a data-driven upsampling network are considered as denoiser and upsampler respectively. Compared with baseline defenses, DUP-Net has three advantages. First, with DUP-Net as a defense, the target model is more robust to white-box adversarial attacks. Second, the statistical outlier removal provides added robustness since it is a non-differentiable denoising operation. Third, the upsampler network can be trained on a small dataset and defends well against adversarial attacks generated from other point cloud datasets. We conduct various experiments to validate that DUP-Net is very effective as defense in practice. Our best defense eliminates 83.8% of C&W and l2 loss based attack (point shifting), 50.0% of C&W and Hausdorff distance loss based attack (point adding) and 9.0% of saliency map based attack (point dropping) under 200 dropped points on PointNet.

Person Search plays the role of the ultimate destination of person re-identification (re-ID) in real applications. It has many challenges that person re-ID doesn't need to handle, such as mis-detections, false alarms and multi-scale matching. In contrast to previous works, we show that a strong multi-scale person matching system can result in a good person search performance with a common deep object detector (e.g. Faster-RCNN). In this work, we provide a robust person search method called Dynamic Hierarchical Feature Fusion (DHFF) which is based on multi-level feature fusion to tackle with multi-scale matching. In addition, A Multi-Metric loss is proposed to train the model effectively and stably with numerous identities. We evaluate our method on two large person search benchmark data sets: CUHK-SYSU and PRW. Experiments show that the proposed algorithm outperforms other state-of-the-art person search methods.

Due to the amazing progresses in deep learning techniques, steganography has now been challenged to tackle not only artificial feature-based but also effective deep-learning-based steganalysis. Recent steganographers have tried to conduct adversarial attacks to defend the steganalysis networks by fine-tuning the embedding details with the help of adversarial information, which, however, mostly are white-box attacks. This research studies a novel method to conduct stegano-graphic adversarial attacks in practical scenario where stegos are sandwiched between black boxes. In our case, the toolboxes to generate stegos are steganographic black boxes where embedding adjustments are prohibited, and networks to detect stegos are semi-black boxes where most of the steganalysis networks' details are unavailable. By reforming few-pixel-attack into the form of extraction conservation noises and add them directly onto stegos, we ensure the message extraction and launch the attack in practical scenario. Experiments show that the proposed method can significantly boost the error rate of the deep-learning-based steganalysis and at the same time keep a comparable error rate when facing artificial feature-based steganalysis.

Crowd counting is a challenging task due to the issues such as scale variation and perspective variation in real crowd scenes. In this paper, we propose a novel Cascaded Residual Density Network (CRDNet) in a coarse-to-fine approach to generate the high-quality density map for crowd counting more accurately. (1) We estimate the residual density maps by multi-scale pyramidal features through cascaded residual density modules. It can improve the quality of density map layer by layer effectively. (2) A novel additional local count loss is presented to refine the accuracy of crowd counting, which reduces the errors of pixel-wise Euclidean loss by restricting the number of people in the local crowd areas. Experiments on two public benchmark datasets show that the proposed method achieves effective improvement compared with the state-of-the-art methods.

Deep neural networks are vulnerable to adversarial examples, subtly perturbed images which can fool networks to output incorrect classification results. To deceive deep learning models, in this paper, instead of utilizing the weakness of networks themselves, we present Embedding Attack, which is to attack the common image resizing operation in the deep learning preprocessing pipeline. By this attack, adversaries can embed a small target image into a benign image to produce adversarial examples without querying the target network. When the adversarial example is resized to the required shape, the embedded target image will be recovered. We design embedding attacks for three common image resizing methods and prove that our algorithms are optimal when the target image can be fully recovered. Furthermore, we design a universal embedding attack that enables adversarial examples to work under different resizing methods.

Video object segmentation is a challenging task with wide variety of applications. Although recent CNN based methods have achieved great performance, they are far from being applicable for real time applications. In this paper, we propose a propagation based video object segmentation method in compressed domain to accelerate inference speed. We only extract features from I-frames by the traditional deep segmentation network. And the features of P-frames are propagated from I-frames. Apart from feature warping, we propose two effective modules in the process of feature propagation to ensure the representation ability of propagated features in terms of appearance and location. Residual supplement module is used to supplement appearance information lost in warping, and spatial attention module mines accurate spatial saliency prior to highlight the specified object. Compared with recent state-of-the-art algorithms, the proposed method achieves comparable accuracy while much faster inference speed.

Recent approaches have achieved great success on still image object detection. Despite the high accuracy, directly applying image object detectors for video object detection is rather slow. Inspired from the fact that object tracking is much more efficient than object detection, we propose to combine object detection and tracking for fast video object detection. Computational expensive detection network is applied on sparsely arranged key frames, while proposals of non-key frames are obtained through tracking and regression of previous frame's proposals. Assisted with an adaptive key-frame arrangement module, our method can adaptively decide whether to track or to detect based on tracking quality. Extensive experiments show that the proposed method can significantly boost detection speed with a rather small drop in detection accuracy.

Screen-camera communication techniques achieve one-way communication with widely-used screen and camera devices. Popular barcode methods use visible spatial patterns to represent data, which has been considered obtrusive to human observers. Recent works borrow ideas from visual light communication (VLC), and use inter-frame pixel change to modulate data. To recover pixel change, the receiver has to record and process video frames. Such video-based extraction has high hardware requirements and lacks reliability. Inspired by VLC-based methods, we propose UnseenCode, an invisible on-screen barcode scheme. It leverages inter-frame embedment from VLC-based methods to embed invisible barcodes into arbitrary on-screen contents. Unlike VLC-based methods, UnseenCode does not require video-based extraction. We propose an image-based extraction method based on cross-component correlation of color images. Any off-the-shelf smartphones with camera capability can be used to read UnseenCode by capturing on-screen contents. We propose the full implementation of UnseenCode for evaluation. Experimental results show that UnseenCode decoding algorithm is reliable and applicable under various screen and camera settings. UnseenCode provides up to 2.5 kbit capacity with less than 5% error rate.

Video object segmentation plays an important role in computer vision and has attracted much attention. Although many recent works have removed the fine-tuning process in pursuit of fast inference speed, while achieving high segmentation accuracy, they are still far from being real-time. In this paper, we regard this task as a feature matching problem and propose a prior probability based metric learning (PPML) method for faster inference speed and higher segmentation accuracy. The proposed method consists of two ingredients: a novel template space updating strategy that improves the efficiency of segmentation by avoiding the explosion of data in template space, and a novel feature matching method which applies more potential probability information through integrating the prior of the first frame and the predicted score of previous frames. Experimental results on DAVIS datasets demonstrate that the proposed method reaches the state-of-the-art competitive performance and is more efficient in time consumption.

Advances in signal processing in the encrypted domain and cloud computing have given rise to privacy‐preserving technologies. In recent years, reversible data hiding in encrypted images (RDH‐EI) has received attention from the research community because additional data can be embedded into an encrypted image without accessing its original content, and the encrypted image can be losslessly recovered after extracting the embedded data. Although the recent development of RDH‐EI compatible with homomorphic public key cryptosystems has intensified research interest, most of the existing mature RDH schemes cannot be transplanted to the encrypted domain due to the limitations of the underlying cryptosystems. In this paper, prediction error expansion based RDH‐ED using probabilistic and homomorphic properties of the Paillier cryptosystem is presented. This work implements non‐integer mean value computation in the encrypted domain without any interactive protocol between the content owner and the cloud server. This work presents mathematical detail of pixel prediction (mean), prediction error, error expansion and data embedding in the encrypted domain and data extraction and content recovery in the plain domain. Experimental results from standard test images reveal that the proposed scheme outperforms other state‐of‐the‐art encrypted domain schemes.

Currently, the most successful model for image adaptive steganography is the framework of minimal distortion, in which a reasonable definition of costs can improve the security level. In the authors' previous work, they developed a rule for cost reassignment in spatial domain called the ‘controversial pixel prior (CPP)’ rule, which defines controversial pixels by utilizing the controversies among several comparable schemes. The CPP rule gives controversial pixels higher modification priorities. In this study, they investigate migrating the CPP rule from the spatial domain to the joint photographic experts group (JPEG) domain and name it the J‐CPP rule. In JPEG images, the cover elements are discrete cosine transform (DCT) coefficients and variant factors mayinfluence the distortion definition includingquantisation step, inter‐blocks correlation and block energy. However, there is no evidence to reveal which factor is of highest priority for promoting security. In this work, they investigate which factor is more helpful in promoting J‐CPP rule, and they finally determine to set the spatial block residual as a penalty to perfect J‐CPP rule. Through extensive experiments on different JPEG steganographic algorithms and steganalysis features, they demonstrate that the J‐CPP rule can improve the security of JPEG adaptive steganography.

With the dramatically increasing deployment of the Internet of Things (IoT), remote monitoring of health data to achieve intelligent healthcare has received great attention recently. However, due to the limited computing power and storage capacity of IoT devices, users' health data are generally stored in a centralized third party, such as the hospital database or cloud, and make users lose control of their health data, which can easily result in privacy leakage and single-point bottleneck. In this paper, we propose Healthchain, a large-scale health data privacy preserving scheme based on blockchain technology, where health data are encrypted to conduct fine-grained access control. Specifically, users can effectively revoke or add authorized doctors by leveraging user transactions for key management. Furthermore, by introducing Healthchain, both IoT data and doctor diagnosis cannot be deleted or tampered with so as to avoid medical disputes. Security analysis and experimental results show that the proposed Healthchain is applicable for smart healthcare system.

Space information network (SIN) makes it possible for any object to be connected to the Internet anywhere, even in the areas with extreme conditions, where a cellular network is not easy to deploy. Access authentication is the key to secure users' access control in SIN, mainly to prevent illegal adversaries from getting access to SIN services. However, the highly complicated communication environment of SIN (e.g., exposed links, higher signal delay, etc.) poses a challenging issue in the design of a secure and efficient authentication scheme. Although some authentication schemes have been proposed for SIN, they are unsuitable for Internet of Things (IoT) in SIN due to the high signaling overhead and insufficient security properties. Therefore, in this paper, we design a provably secure and efficient authentication protocol, along with an efficient handover mechanism, for IoT in SIN. In our design, we introduce a new authentication system model, where the satellites are given the ability to authenticate users to avoid the online involvement of the network control center (NCC) when authenticating users, thereby reducing long authentication delay and avoiding a single point of bottleneck in NCC. Furthermore, the support of batch verification in our design can significantly enhance handover efficiency when a group of users switch to another satellite. Our further analysis shows that our scheme is secure against various attacks and can meet a variety of security requirements. In addition, performance evaluation shows the superiority of our scheme on both delay and handover efficiency compared with existing schemes.

The ability of the light-emitting diodes (LEDs) on a keyboard to send data at a rate that is far greater than the human eye can perceive has been fully studied. However, an IP camera can only fetch high-resolution images at a low frame rate. It is unable to act as a sink of the optical covert channel directly. In this paper, a novel form of modulation is proposed to modulate the LEDs on a keyboard. The modulated signal can be received by a nearby IP camera. To verify its validity, we implement a prototype of exfiltration malware. Our experiment shows a significant improvement in the imperceptibility of covert communication. Against the background of the Internet of Things, it is possible to leak data covertly to IP cameras across air-gapped networks via LED keyboard status indicators.

On-body wireless networks (oBWNs) play a crucial role in improving the ubiquitous healthcare services. Using oBWNs, the vital physiological information of the patient can be gathered from the wearable sensor nodes and accessed by the authorized user like the health professional or the doctor. Since the open nature of wireless communication and the sensitivity of physiological information, secure communication has always been the vital issue in oBWNs-based systems. In recent years, several authentication schemes have been proposed for remote patient monitoring. However, most of these schemes are so susceptible to security threats and not suitable for practical use. Specifically, all these schemes using lightweight cryptographic primitives fail to provide forward secrecy and suffer from the desynchronization attack. To overcome the historical security problems, in this paper, we present a lightweight and secure three-factor authentication scheme for remote patient monitoring using oBWNs. The proposed scheme adopts one-time hash chain technique to ensure forward secrecy, and the pseudonym identity method is employed to provide user anonymity and resist against desynchronization attack. The formal and informal security analyses demonstrate that the proposed scheme not only overcomes the security weaknesses in previous schemes but also provides more excellent security and functional features. The comparisons with six state-of-the-art schemes indicate that the proposed scheme is practical with acceptable computational and communication efficiency.

Locally decodable codes and locally correctable codes (LCCs) have several important applications, such as private information retrieval, secure multiparty computation, and circuit lower bounds. Three major parameters are considered in LCCs: query complexity, message length, and codeword length. The most familiar LCCs in the regime of low query complexity are the generalized Reed-Muller (GRM) codes. However, it has not previously been determined whether there exist codes that have shorter codeword lengths than GRM codes with the same query complexity and message length. In this paper, we show that the projective Reed-Muller (PRM) codes are such LCCs for some parameters. The GRM code is specified by the alphabet size q, the number of variables m, and the degree d, where d ≤ q - 2. When d = q - 2 and q - 1 is a power of a prime, we prove that there exists a PRM code with shorter codeword length than the GRM code with the same query complexity and message length. We also present for these PRM codes a perfectly smooth local decoder to recover a symbol in a codeword by accessing not more than q symbols at the coordinates of the codeword.

Different from all the previous reversible data hiding schemes, a completely novel one for the color image is proposed, which reversibly embeds messages into the color host image without modifying its corresponding gray version. The property of grayscale invariance is valuable, because many applications and image processing algorithms for color images are based on the corresponding gray versions, such as black and white printing, producing reading materials for color blind people, single-channel image processing, and so on. Thus, in terms of these applications and image processing algorithms, the presented scheme will make the generated color marked image be free for its further uses. In this paper, the unchanged gray version is utilized efficiently in both the embedding processes and the extracting processes. Messages are embedded into the red and blue channels of color image, and then the green channel is adjusted adaptively to remove the offsets from the gray version caused by modifying its red and blue channels. To return the adjusted green channel, error correcting bits guaranteeing the reversibility are regarded as one part of payloads to be recursively embedded. Therefore, the reversibility and the property of grayscale invariance are both achieved.

Minimal distortion steganography is the most successful model for adaptive steganography, in which the cost function determines the security. Texture complexity is the major factor in defining cost function in images. In this paper, we proposed a method to improve the cost function of JPEG steganography by exploiting the texture in microscale. The proposed scheme is designed by using a “microscope” to highlight details in an image, so that distortion definition can be more refined. Linear unsharp masking acts as the microscope, because it can accentuate the texture region as well as maintain the original characteristics of images. Inter-block spreading rule is proposed to further strengthen the security. We improve the state-of-the-art schemes, J-UNIWARD and UERD, as J-UNIWARD has outstanding performance on resisting detection while UERD has significant lower computational complexity. In order to keep high efficiency of UERD, filtering in the DCT domain is introduced. Extending experiments show that in most cases the proposed methods (J-MSUNIWARD and MSUERD) can achieve a higher level of security than the original methods.

This paper proposes a novel screen-shooting resilient watermarking scheme, which means that if the watermarked image is displayed on the screen and the screen information is captured by the camera, we can still extract the watermark message from the captured photo. To realize such demands, we analyzed the special distortions caused by the screen-shooting process, including lens distortion, light source distortion, and moiré distortion. To resist the geometric deformation caused by lens distortion, we proposed an intensity-based scale-invariant feature transform (I-SIFT) algorithm which can accurately locate the embedding regions. As for the loss of image details caused by light source distortion and moiré distortion, we put forward a small-size template algorithm to repeatedly embed the watermark into different regions, so that at least one complete information region can survive from distortions. At the extraction side, we designed a cross-validation-based extraction algorithm to cope with repeated embedding. The validity and correctness of the extraction method are verified by hypothesis testing. Furthermore, to boost the extraction speed, we proposed a SIFT feature editing algorithm to enhance the intensity of the keypoints, based on which, the extraction accuracy and extraction speed can be greatly improved. The experimental results show that the proposed watermarking scheme achieves high robustness for screen-shooting process. Compared with the previous schemes, our algorithm provides significant improvement in robustness for screen-shooting process and extraction efficiency.

Nowadays, the Space Information Network (SIN) has been widely used in real life because of its advantages of communicating anywhere at any time. This feature is leading to a new trend that traditional wireless users are willing to roam to SIN to obtain a better service. However, the features of exposed links and higher signal latency in SIN make it difficult to design a secure and fast roaming authentication scheme for this new trend. Although some existing researches have been focused on designing secure authentication protocols for SIN or providing roaming authentication protocols for traditional wireless networks, these schemes cannot provide adequate requirements for the roaming communication in SIN and bring in critical issues, such as the privacy leakage or intolerable authentication delay. Observing these problems have not been well addressed, we design an anonymous and fast roaming authentication scheme for SIN. In our scheme, we utilize the group signature to provide the anonymity for roaming users, and assume that the satellites have limited computing capacity and make them have the defined authentication function to avoid the real-time involvement of the home network control center when authenticating the roaming users. The results of security and performance analysis show that the proposed scheme can provide the required security features, while providing a small authentication delay.

We propose a novel technique for steganography on 3-D meshes so as to resist steganalysis. The majority of existing methods modulate vertex coordinates to embed messages in a nonadaptive way. We take account of complexity of local regions as joint distortion of a triple unit (vertice) and coding method such as syndrome trellis codes to adaptively embed messages, which owns stronger security with respect to existing steganalysis. Key to the distortion is a novel formulation of adaptive steganography, which relies on some effective steganalytic features such as variation of vertex normal. We provide quantitative and qualitative comparisons of our method with several baselines against steganalytic features LFS64, LFS76, and ensemble classifiers, and show that it outperforms the current state of the art. Meanwhile, we proposed an attacking method on steganography proposed by Chao et al. (2009) with a high detection rate.

Due to its large storage facility and high-end computing capability, cloud computing has received great attention as a huge amount of personal multimedia data and computationally expensive tasks can be outsourced to the cloud. However, the cloud being third-party semi-trusted, is prone to information leakage, raising privacy risks. Signal processing in the encrypted domain has emerged as a new research paradigm on privacy-preserving processing over outsourced data by semi-trusted cloud. In this article, we propose a solution for non-integer mean value computation in the homomorphic encrypted domain without any interactive protocol between the client and the service provider. Using the proposed solution, various image processing operations, such as local smoothing filter, un-sharp masking, and histogram equalization, can be performed in the encrypted domain at the cloud server without any privacy concerns. Our experimental results from standard test images reveal that these image processing operations can be performed without pre-processing, without client-server interactive protocol, and without any error between the encrypted domain and the plain domain.

This paper presents the first attempt at stereoscopic neural style transfer, which responds to the emerging demand for 3D movies or AR/VR. We start with a careful examination of applying existing monocular style transfer methods to left and right views of stereoscopic images separately. This reveals that the original disparity consistency cannot be well preserved in the final stylization results, which causes 3D fatigue to the viewers. To address this issue, we incorporate a new disparity loss into the widely adopted style loss function by enforcing the bidirectional disparity constraint in non-occluded regions. For a practical realtime solution, we propose the first feed-forward network by jointly training a stylization sub-network and a disparity sub-network, and integrate them in a feature level middle domain. Our disparity sub-network is also the first end-to-end network for simultaneous bidirectional disparity and occlusion mask estimation. Finally, our network is effectively extended to stereoscopic videos, by considering both temporal coherence and disparity consistency. We will show that the proposed method clearly outperforms the baseline algorithms both quantitatively and qualitatively.

Detecting and localizing anomalies in surveillance videos is an ongoing challenge. Most existing methods are patch or trajectory-based, which lack semantic understanding of scenes and may split targets into pieces. To handle this problem, this paper proposes a novel and effective algorithm by incorporating deep object detection and tracking with full utilization of spatial and temporal information. We propose a new dynamic image by fusing both appearance and motion information and feed it into object detection network, which can detect and classify objects precisely even in dim and crowd scenes. Based on the detected objects, we develop an effective and scale-insensitive feature, named histogram variance of optical flow angle (HVOFA), together with motion energy to find abnormal motion patterns. In order to further discover missing anomalies and reduce false detected ones, we conduct a post-processing step with abnormal object tracking. The proposed algorithm outperforms state-of-the-art methods on standard benchmarks.

In prevalent two-stage object detectors, ROI pooling or position sensitive ROI pooling (PS ROI pooling) is usually used to extract features of proposal. But ROI pooling or PS ROI pooling ignores the local or global information of proposal respectively. It motivates us to design a kind of pooling method which can capture both global information and local information of proposal. In this paper, we propose pyramid sub-region sensitive network (PSSNet) for object detection which uses pyramid sub-region sensitive ROI pooling (PSS ROI pooling) to extract features of proposal. The PSS ROI pooling can capture the global and coarse-to-fine local information of proposal. Then, we explore different weighting strategies to utilize the PSS ROI features using self-adapting learning factors. Our PSSNet achieves the state-of-art result on PASCAL VOC 2007, PASCAL VOC 2012 datasets and competitive result on MS COCO dataset.

How to effectively utilize the temporal information in video has been an important problem in visual tracking. In this paper, we try to address this problem from two aspects. At first, we use optical flow to take advantage of the inter-frame information, when we obtain the position of the last frame, we predict the approximate location in present frame by calculating the optical flow and generate samples around it. Secondly, we use the tracked patches as reference to identify the target better. We designed a Siamese network which take image pairs consisted of exemplars and samples as inputs, the objective function is also modified by adding a priori probability. Further, we conducted experiments on the OTB benchmark and achieve competitive result both on accuracy and robustness, which demonstrate the effectiveness of our proposed algorithm.

Many artificial intelligence tasks appear in dual forms like English$\leftrightarrow$French translation and speech$\leftrightarrow$text transformation. Existing dual learning schemes, which are proposed to solve a pair of such dual tasks, explore how to leverage such dualities from data level. In this work, we propose a new learning framework, model-level dual learning, which takes duality of tasks into consideration while designing the architectures for the primal/dual models, and ties the model parameters that playing similar roles in the two tasks. We study both symmetric and asymmetric model-level dual learning. Our algorithms achieve significant improvements on neural machine translation and sentiment analysis.

Recent studies have shown that the non-additive distortion model of Decomposing Joint Distortion ($DeJoin$) can work well for spatial image steganography by defining joint distortion with the principle of Synchronizing Modification Directions (SMD). However, no principles have yet produced to instruct the definition of joint distortion for JPEG steganography. Experimental results indicate that SMD can not be directly used for JPEG images, which means that simply pursuing modification directions clustered does not help improve the steganographic security. In this paper, we inspect the embedding change from the spatial domain and propose a principle of Block Boundary Continuity (BBC) for defining JPEG joint distortion, which aims to restrain blocking artifacts caused by inter-block adjacent modifications and thus effectively preserve the spatial continuity at block boundaries. According to BBC, whether inter-block adjacent modifications should be synchronized or desynchronized is related to the DCT mode and the adjacent direction of inter-block coefficients (horizontal or vertical). When built into $DeJoin$, experiments demonstrate that BBC does help improve state-of-the-art additive distortion schemes in terms of relatively large embedding payloads against modern JPEG steganalyzers.

Deep neural network based steganalysis has developed rapidly in recent years, which poses a challenge to the security of steganography. However, there is no steganography method that can effectively resist the neural networks for steganalysis at present. In this paper, we propose a new strategy that constructs enhanced covers against neural networks with the technique of adversarial examples. The enhanced covers and their corresponding stegos are most likely to be judged as covers by the networks. Besides, we use both deep neural network based steganalysis and high-dimensional feature classifiers to evaluate the performance of steganography and propose a new comprehensive security criterion. We also make a tradeoff between the two analysis systems and improve the comprehensive security. The effectiveness of the proposed scheme is verified with the evidence obtained from the experiments on the BOSSbase using the steganography algorithm of WOW and popular steganalyzers with rich models and three state-of-the-art neural networks.

With an advancement of mobile communication technology, the space information network (SIN) has been proposed to meet the increasing demands of mobile communication due to its advantage of providing great expanding access services. In SIN, authentication is significant for the security to prevent the network resource from unauthorized access. However, the features of highly exposed links and extremely high propagation delay make it difficult to design a secure and fast authentication scheme for SIN. Although some existing researches have tried to design authentication protocols for SIN, they haven't taken the intolerable authentication delay and the risk of satellite compromising into consideration. Faced with these problems, we design a proxy signature-based authentication scheme for SIN, in which, the interaction process of authentication can be only implemented between the mobile user and the satellite node, thus reducing the long authentication implementation delay. Furthermore, we utilize the proxy signature to mitigate the risk of satellites being attacked. The results of security and performance analysis show that the proposed scheme can provide the required security and largely reduce the authentication latency.

Anomaly detection in crowded scenes is an important issue in computer vision. In this paper, we propose a novel framework which takes both appearance and motion characteristics into consideration to detect anomalies. A new foreground object localization method is put forward at first to extract object proposals. For motion representation, we present a novel local motion based descriptor named as Spatially Localized Multi-scale Histogram of Optical Flow (SL-MHOF) to capture the local motion statistics for each object proposal. For appearance representation, we apply convolutional neural networks (CNNs) because of their high visual discriminative capacities. These two features are then fed into Gaussian Mixture Model (GMM) Classifiers respectively to generate anomaly scores, which are fused with a softmax function to produce the final anomaly detection results. Experiments on UCSD datasets indicate the effectiveness of our proposed approach, which achieves state-of-the-art performance.

We propose a new model-free approach for crowd tracking that integrates low-level keypoint tracking, midlevel patch tracking, and high-level group evolution in one unified framework. Instead of computing optical flows, tracking keypoints, or pedestrians, we propose to represent the crowd as a set of distinctive and stable midlevel patches. These patches are tracked together through occlusions, background clutter, and appearance variations, with spatial relations modeled by the proposed hierarchical tree structure. In the low level, keypoint tracking provides accurate local motions, which guides the detection of midlevel patches with stable internal motions, and also organizes patches into hierarchical groups with collective motions. In the high level, group evolution guides updating of the proposed hierarchical tree structure through merge and split events. The dynamically structured patches not only substantially improve their own tracking, but also act as assistant patches that can help track given targets more accurately in a crowd. Extensive experiments on both ours and publicly available data sets show that our proposed approach significantly outperforms current state-of-the-art trackers.

Recursive code construction (RCC), based on the optimal transition probability matrix (OTPM), approaching the rate-distortion bound of reversible data hiding (RDH) has been proposed. Using the existing methods, OTPM can be effectively estimated only for a consistent distortion metric, i.e., if the host elements at different positions share the same distortion metric. However, in many applications, the distortion metrics are position dependent and should thus be inconsistent. Inconsistent distortion metrics can usually be quantified as a multi-distortion metric. In this paper, we first formulate the rate-distortion problem of RDH under a multi-distortion metric and subsequently propose a general framework to estimate the corresponding OTPM, with which RCC is extended to approach the rate-distortion bound of RDH under the multi-distortion metric. We apply the proposed framework to two examples of inconsistent distortion metrics: RDH in color image and reversible steganography. The experimental results show that the proposed method can efficiently improve upon the existing techniques.

Reversible data hiding in encrypted domain (RDH-ED) has greatly attracted researchers as the original content can be losslessly reconstructed after the embedded data are extracted, while the content owner's privacy remains protected. Most of the existing RDH-ED algorithms are designed for grayscale/color images, which cannot be directly applied to other carriers, such as three-dimensional (3D) meshes. With the rapid development of 3D related applications, 3D models have been widely used on the Internet, which motivated us to design a reliable RDH-ED scheme for 3D meshes. The proposed method maps decimals of the vertex coordinates into integers first, so that a bit-stream encryption technique can be executed. With a data-hiding key, several least-significant bits are operated to embed data. By using the encryption key, a receiver can roughly reconstruct the content of the mesh. According to the data-hiding key, with the aid of spatial correlation in natural mesh models, the embedded data can be successfully extracted and the original mesh can be perfectly recovered. Experiments show that the proposed method has a high data-embedding payload, maintains high values of the decrypted meshes, and has low computational complexity.

We propose StyleBank, which is composed of multiple convolution filter banks and each filter bank explicitly represents one style, for neural image style transfer. To transfer an image to a specific style, the corresponding filter bank is operated on top of the intermediate feature embedding produced by a single auto-encoder. The StyleBank and the auto-encoder are jointly learnt, where the learning is conducted in such a way that the auto-encoder does not encode any style information thanks to the flexibility introduced by the explicit filter bank representation. It also enables us to conduct incremental learning to add a new image style by learning a new filter bank while holding the auto-encoder fixed. The explicit style representation along with the flexible network design enables us to fuse styles at not only the image level, but also the region level. Our method is the first style transfer network that links back to traditional texton mapping methods, and hence provides new understanding on neural style transfer. Our method is easy to train, runs in real-time, and produces results that qualitatively better or at least comparable to existing methods.

Multi-label image classification is a fundamental but challenging task in computer vision. Great progress has been achieved by exploiting semantic relations between labels in recent years. However, conventional approaches are unable to model the underlying spatial relations between labels in multi-label images, because spatial annotations of the labels are generally not provided. In this paper, we propose a unified deep neural network that exploits both semantic and spatial relations between labels with only image-level supervisions. Given a multi-label image, our proposed Spatial Regularization Network (SRN) generates attention maps for all labels and captures the underlying relations between them via learnable convolutions. By aggregating the regularized classification results with original results by a ResNet-101 network, the classification performance can be consistently improved. The whole deep neural network is trained end-to-end with only image-level annotations, thus requires no additional efforts on image annotations. Extensive evaluations on 3 public datasets with different types of labels show that our approach significantly outperforms state-of-the-arts and has strong generalization capability. Analysis of the learned SRN model demonstrates that it can effectively capture both semantic and spatial relations of labels for improving classification performance.

Recently, CNN (Convolutional Neural Network) based trackers have achieved promising results benefited from their robust feature representation. However, most trackers only use features from a certain layer, which limits their performance. In this paper, we propose a novel CNN based tracker. Firstly, we use local detection and global detection network for target localization. In local detection network, we fuse features from different layers to train a fully convolutional neural network for target localization. In case the local detection network fails when the target disappear for a while and appears in another location, we train a global detection network to detect if the target appears again. Then, we employ a correlation filter to estimate accurate scale of the target using HOG features extracted around predicted location. Extensive experiments on various challenging video sequences demonstrate the effectiveness of our proposed algorithm compared with several state-of-the-art trackers.

Training a feed-forward network for the fast neural style transfer of images has proven successful, but the naive extension of processing videos frame by frame is prone to producing flickering results. We propose the first end-to-end network for online video style transfer, which generates temporally coherent stylized video sequences in near realtime. Two key ideas include an efficient network by incorporating short-term coherence, and propagating short-term coherence to long-term, which ensures consistency over a longer period of time. Our network can incorporate different image stylization networks and clearly outperforms the per-frame baseline both qualitatively and quantitatively. Moreover, it can achieve visually comparable coherence to optimization-based video style transfer, but is three orders of magnitude faster.

In this paper, we propose a CNN-based framework for online MOT. This framework utilizes the merits of single object trackers in adapting appearance models and searching for target in the next frame. Simply applying single object tracker for MOT will encounter the problem in computational efficiency and drifted results caused by occlusion. Our framework achieves computational efficiency by sharing features and using ROI-Pooling to obtain individual features for each target. Some online learned target-specific CNN layers are used for adapting the appearance model for each target. In the framework, we introduce spatial-temporal attention mechanism (STAM) to handle the drift caused by occlusion and interaction among targets. The visibility map of the target is learned and used for inferring the spatial attention map. The spatial attention map is then applied to weight the features. Besides, the occlusion status can be estimated from the visibility map, which controls the online updating process via weighted loss on training samples with different occlusion statuses in different frames. It can be considered as temporal attention mechanism. The proposed algorithm achieves 34.3% and 46.0% in MOTA on challenging MOT15 and MOT16 benchmark dataset respectively.

Convolutional neural network (CNN) is inspired by the biological structure of human visual system (HVS). And there are still mechanisms in HVS that are worthy learn from. We were inspired by the function of feature selection in HVS which is named as visual saliency and proposed Sal-Mask connection and Ada-Sal Network to implement similar function in neural networks. In this paper, we did further research and tried three different improvement schemes on Ada-Sal Network. By visualizing features we illustrated the feature selection ability of Sal-Mask connection helps the neural network to extract features from more subtle details of input images. The experiment results proved that neural network works better with this saliency based feature selection function. We also found that Sal-Mask connection works best on features from the first convolutional connection.

Many supervised learning tasks are emerged in dual forms, e.g., English-to-French translation vs. French-to-English translation, speech recognition vs. text to speech, and image classification vs. image generation. Two dual tasks have intrinsic connections with each other due to the probabilistic correlation between their models. This connection is, however, not effectively utilized today, since people usually train the models of two dual tasks separately and independently. In this work, we propose training the models of two dual tasks simultaneously, and explicitly exploiting the probabilistic correlation between them to regularize the training process. For ease of reference, we call the proposed approach dual supervised learning. We demonstrate that dual supervised learning can improve the practical performances of both tasks, for various applications including machine translation, image processing, and sentiment analysis.

With the fast development of deep learning, it has become common to learn big neural networks using massive training data. Asynchronous Stochastic Gradient Descent (ASGD) is widely adopted to fulfill this task for its efficiency, which is, however, known to suffer from the problem of delayed gradients. That is, when a local worker adds its gradient to the global model, the global model may have been updated by other workers and this gradient becomes “delayed”. We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD. This is achieved by leveraging Taylor expansion of the gradient function and efficient approximators to the Hessian matrix of the loss function. We call the new algorithm Delay Compensated ASGD (DC-ASGD). We evaluated the proposed algorithm on CIFAR-10 and ImageNet datasets, and the experimental results demonstrate that DC-ASGD outperforms both synchronous SGD and asynchronous SGD, and nearly approaches the performance of sequential SGD.

Recent years have witnessed the rapid development of machine learning in solving artificial intelligence (AI) tasks in many domains, including translation, speech, image, etc. Within these domains, AI tasks are usually not independent. As a specific type of relationship, structural duality does exist between many pairs of AI tasks, such as translation from one language to another vs. its opposite direction, speech recognition vs. speech synthetization, image classification vs. image generation, etc. The importance of such duality has been magnified by some recent studies, which revealed that it can boost the learning of two tasks in the dual form. However, there has been little investigation on how to leverage this invaluable relationship into the inference stage of AI tasks. In this paper, we propose a general framework of dual inference which can take advantage of both existing models from two dual tasks, without re-training, to conduct inference for one individual task. Empirical studies on three pairs of specific dual tasks, including machine translation, sentiment analysis, and image processing have illustrated that dual inference can significantly improve the performance of each of individual tasks.

The encoder-decoder framework has achieved promising progress for many sequence generation tasks, including machine translation, text summarization, dialog system, image captioning, etc. Such a framework adopts an one-pass forward process while decoding and generating a sequence, but lacks the deliberation process: A generated sequence is directly used as final output without further polishing. However, deliberation is a common behavior in human's daily life like reading news and writing papers/articles/books. In this work, we introduce the deliberation process into the encoder-decoder framework and propose deliberation networks for sequence generation. A deliberation network has two levels of decoders, where the first-pass decoder generates a raw sequence and the second-pass decoder polishes and refines the raw sentence with deliberation. Since the second-pass deliberation decoder has global information about what the sequence to be generated might be, it has the potential to generate a better sequence by looking into future words in the raw sentence. Experiments on neural machine translation and text summarization demonstrate the effectiveness of the proposed deliberation networks. On the WMT 2014 English-to-French translation task, our model establishes a new state-of-the-art BLEU score of 41.5.

In the past decade, golf has stimulated people’s great interest and the number of golf players has increased significantly. Therefore, how to train a golfer to make a perfect swing has attracted extensive research attentions. Among these researches, the most important step is to capture and reconstruct the swing movement in a transportable and non-intrusive way. Restricted by the development of present depth imaging devices, the initial captured swing movement may not be acceptable due to occlusions and mixing up of body parts. In this paper, to restore motion information from self-occlusion and reconstruct 3D golf swing from low resolution data, a Dynamic Bayesian Network (DBN) model based golf swing reconstruction algorithm is proposed to increase the capture accuracy considering the spatial and temporal similarities of swing between different golfers. A Smart Motion Reconstruction system for Golf swing, SMRG, is presented based on the DBN model with a popular depth imaging device, Kinect, as capturing device. Experimental results have proved that the proposed system can achieve comparable reconstruction accuracy to the commercial optical motion caption (OMocap) system and better performance than state of art modification algorithms using depth information.

Online image sharing in social platforms can lead to undesired privacy disclosure. For example, some enterprises may detect these large volumes of uploaded images to do users’ in-depth preference analysis for commercial purposes. And their technology might be today’s most powerful learning model, deep neural network (DNN). To just elude these automatic DNN detectors without affecting visual quality of human eyes, we design and implement a novel Stealth algorithm, which makes the automatic detector blind to the existence of objects in an image, by crafting a kind of adversarial examples. It is just like all objects disappear after wearing an “invisible cloak” from the view of the detector. Then we evaluate the effectiveness of Stealth algorithm through our newly defined measurement, named privacy insurance. The results indicate that our scheme has considerable success rate to guarantee privacy compared with other methods, such as mosaic, blur, and noise. Better still, Stealth algorithm has the smallest impact on image visual quality. Meanwhile, we set a user adjustable parameter called cloak thickness for regulating the perturbation intensity. Furthermore, we find that the processed images have transferability property; that is, the adversarial images generated for one particular DNN will influence the others as well.

Attribute-based encryption (ABE) has opened up a popular research topic in cryptography over the past few years. It can be used in various circumstances, as it provides a flexible way to conduct fine-grained data access control. Despite its great advantages in data access control, current ABE based access control system cannot satisfy the requirement well when the system judges the access behavior according to attribute comparison, such as “greater than x” or “less than x”, which are called comparable attributes in this paper. In this paper, based on a set of well-designed sub-attributes representing each comparable attribute, we construct a comparable attribute-based encryption scheme (CABE for short) to address the aforementioned problem. The novelty lies in that we provide a more efficient construction based on the generation and management of the sub-attributes with the notion of 0-encoding and 1-encoding. Extensive analysis shows that: Compared with the existing schemes, our scheme drastically decreases the storage, communication and computation overheads, and thus is more efficient in dealing with the applications with comparable attributes.

Recent advances on adaptive steganography imply that the security of steganography can be improved by exploiting the mutual impact of modifications between adjacent cover elements, such as pixels of images, which is called a nonadditive distortion model. In this paper, we propose a framework for nonadditive distortion steganography by defining joint distortion on pixel blocks. To reduce the complexity for minimizing joint distortion, we design a coding method to decompose the joint distortion (abbreviated to DeJoin) into distortion on individual pixels; thus, the message can be efficiently embedded with syndrome-trellis codes. We prove that DeJoin can approach the lower bound of joint distortion. As an example, we define joint distortion according to the principle of synchronizing modification direction and then design steganographic algorithms with DeJoin. The experimental results show that the proposed method outperforms previous nonadditive distortion steganography when resisting the state-of-the-art steganalysis.

Industries and individuals outsource database to realize convenient and low-cost applications and services. In order to provide sufficient functionality for SQL queries, many secure database schemes have been proposed. However, such schemes are vulnerable to privacy leakage to cloud server. The main reason is that database is hosted and processed in cloud server, which is beyond the control of data owners. For the numerical range query (“>,” “<;,” and so on), those schemes cannot provide sufficient privacy protection against practical challenges, e.g., privacy leakage of statistical properties, access pattern. Furthermore, increased number of queries will inevitably leak more information to the cloud server. In this paper, we propose a two-cloud architecture for secure database, with a series of intersection protocols that provide privacy preservation to various numeric-related range queries. Security analysis shows that privacy of numerical information is strongly protected against cloud providers in our proposed scheme.

In steganography schemes, the distortion function is used to define modification costs on cover elements, which is distinctly vital to the security of modern adaptive steganography. There are several successful rules for reassigning the costs defined by a given distortion function, which can promote the security level of the corresponding steganographic algorithm. In this paper, we propose a novel cost reassignment rule, which is applied to not one but a batch of existing distortion functions. We find that the costs assigned on some pixels by several steganographic methods may be very different even though these methods exhibit close security levels. We call such pixels “controversial pixel”. Experimental results show that steganalysis features are not sensitive to controversial pixels; therefore, these pixels are suitable to carry more payloads. We name this rule the controversial pixels prior (CPP) rule. Following the rule, we propose a cost reassignment scheme. Through extensive experiments on several kinds of stego algorithms, steganalysis features, and cover databases, we demonstrate that the CPP rule can improve the security of the state-of-the-art steganographic algorithms for spatial images.

Message hiding in texture image synthesis is a novel steganography approach by which we resample a smaller texture image and synthesize a new texture image with a similar local appearance and an arbitrary size. However, the mirror operation over the image boundary is flawed and is easy to attack. We propose an attacking method on this steganography, which can not only detect the stego-images but can also extract the hidden messages.

Feature selection (FS) is an important technique in machine learning and data mining, especially for large-scale high-dimensional data. Most existing studies have been restricted to batch learning, which is often inefficient and poorly scalable when handling big data in real world. As real data may arrive sequentially and continuously, batch learning has to retrain the model for the new coming data, which is very computationally intensive. Online feature selection (OFS) is a promising new paradigm that is more efficient and scalable than batch learning algorithms. However, existing online algorithms usually fall short in their inferior efficacy. In this article, we present a novel second-order OFS algorithm that is simple yet effective, very fast and extremely scalable to deal with large-scale ultra-high dimensional sparse data streams. The basic idea is to exploit the second-order information to choose the subset of important features with high confidence weights. Unlike existing OFS methods that often suffer from extra high computational cost, we devise a novel algorithm with a MaxHeap-based approach, which is not only more effective than the existing first-order algorithms, but also significantly more efficient and scalable. Our extensive experiments validated that the proposed technique achieves highly competitive accuracy as compared with state-of-the-art batch FS methods, meanwhile it consumes significantly less computational cost that is orders of magnitude lower. Impressively, on a billion-scale synthetic dataset (1-billion dimensions, 1-billion non-zero features, and 1-million samples), the proposed algorithm takes less than 3 minutes to run on a single PC.

Recently, graph ranking-based methods have been introduced to visual tracking and achieved promising results due to the local structure preserving property. However, existing graph ranking-based trackers use holistic templates to construct the graphs which makes the trackers sensitive to occlusions. In this paper, we propose a part-based multi-graph ranking algorithm for robust visual tracking. In our method, template samples are divided into local parts. Multiple graphs are constructed based on different part samples and different feature representations. Then, the multiple graphs are integrated into a regularization framework with each graph assigned a weight. Furthermore, by imposing the l2,1 norm on the weight matrix of graphs, the confident parts are selected to reduce the effects of occluded ones. An effective optimization scheme is proposed to learn the weight matrix and the rank scores jointly. Experimental results on various challenging video sequences demonstrate our proposed algorithm outperforms state-of-the-art trackers.

Associating groups of people across non-overlapping camera views is an important but unsolved problem. Compared with the similar person re-identification task, group re-identification introduces some new challenges, such as significant deformation in uncontrolled directions, great intra-group occlusions and so on. In this paper, we propose a novel patch matching based framework for group re-identification. Discriminative salience channels are learned to filter out highly unreliable and non-informative patch matches between two group images, while retain true matches undergoing appearance variations. The resulting candidate correspondences are further explored by the proposed consistent matching process, which prefers coherent matches in true group image pairs. The effectiveness of our approach is validated on two group re-identification datasets: ZeCSS and i-LIDS MCTS. It outperforms state-of-the-art methods on both datasets.

This paper presents a framework for anomaly detection in videos which considers both motion and appearance features. For motion cues, we propose a new feature called 3D-HOF, which effectively extracts both velocity and orientation from the optical flow map. At the same time, we introduce the concept of “depth of field” problem to make the detection more accurate when the velocity of an object may seem to be different according to its distance to the camera. For appearance cues, we use 3D gradients of spatio-temporal cuboids as features. Next in the detection part, we propose a fast double sparse representation method in order to make the process faster in some actual scenes. Finally, we integrate both the outcomes of using motion and appearance cues as the final outcomes. Results on discriminative datasets show efficiency and effectiveness compared to the state-of-the-art methods.

While neural machine translation (NMT) is making good progress in the past two years, tens of millions of bilingual sentence pairs are needed for its training. However, human labeling is very costly. To tackle this training data bottleneck, we develop a dual-learning mechanism, which can enable an NMT system to automatically learn from unlabeled data through a dual-learning game. This mechanism is inspired by the following observation: any machine translation task has a dual task, e.g., English-to-French translation (primal) versus French-to-English translation (dual); the primal and dual tasks can form a closed loop, and generate informative feedback signals to train the translation models, even if without the involvement of a human labeler. In the dual-learning mechanism, we use one agent to represent the model for the primal task and the other agent to represent the model for the dual task, then ask them to teach each other through a reinforcement learning process. Based on the feedback signals generated during this process (e.g., the language-model likelihood of the output of a model, and the reconstruction error of the original sentence after the primal and dual translations), we can iteratively update the two models until convergence (e.g., using the policy gradient methods). We call the corresponding approach to neural machine translation \emph{dual-NMT}. Experiments show that dual-NMT works very well on English$\leftrightarrow$French translation; especially, by learning from monolingual data (with 10\% bilingual data for warm start), it achieves a comparable accuracy to NMT trained from the full bilingual data for the French-to-English translation task.