Transformer inference based on large multimodal models (LMM) has been extensively adopted in ubiquitous intelligence applications due to its exceptional abilities across various domains. However, this emerging service paradigm also exposes a huge attacking surface, making it more susceptible to the risk of sensitive content and model exposure. While steady progress has been made recently in enabling secure inference, existing studies mainly focus on the inference service for singlemodal data. Moreover, it remains underexplored how to achieve an accuracy-efficiency trade-off while ensuring adaptability in secure inference service. In this work, we present SecLMM, the first verifiable service tailored for secure and lightweight LMM-based inference on resource-constrained servers. SecLMM is built from a delicate synergy of techniques from secure multi-party computation (MPC) and LMM domains. To support secure LMM-based inference, we propose to convert the activation functions into MPCfriendly functions by analyzing the derivative, eliminating the need for retraining. To address multimodal data heterogeneity, we explore a secure vectorization scheme to process multimodal data. Besides, we adopt an effective lightweight scheme to remove redundant weights, effectively reducing deployment overhead. Moreover, we integrate a validation scheme based on multimodal mixed datasets to further enhance the reliability of the inference service. Evaluations show that SecLMM can drive full-content protection while maintaining LMM-based inference accuracy comparable to the plaintext domain.

Sample alignment is recognized as a vital component of vertical federated learning, which facilitates the integration of differential samples and high-quality model training. In this trend, providing Private Sample Alignment (PSA) among multi-clients becomes naturally necessary for preventing unauthorized sample access and client privacy exposure. However, exiting PSA protocols mainly focus on two-party scenarios and cannot be directly adapted to the multi-client delegated computing scenarios required for vertical federated learning. Besides, these studies fail to address the need for protocol robustness in practical federated Learning network environments. Therefore, we aim to design an efficient and reliable PSA protocol in multi-client vertical federated learning. In this work, we present the first practical PSA protocol for vertical federated learning, allowing multi-clients to efficiently identify common samples without revealing additional information. Toward this direction, our PSA protocol first explores the Learning With Errors (LWE) problem to create a lightweight delegated Private Set Intersection (PSI) scheme, enabling efficient sample intersection among multiple clients. To achieve the reliability of the PSA protocol, we devise a multi-client vector aggregation algorithm that securely delegates the server to calculate the sample intersection. Building on this foundation, we develop an efficient Threshold-based Private Sample Alignment (T-PSA) protocol that allows multiple clients to determine the intersection of their input samples only if the intersection size surpasses a specific threshold. We implement a prototype and conduct a thorough security analysis. Comprehensive evaluation results confirm the efficiency and practicality of our design.

Federated Learning (FL) transforms collaborative machine learning by enabling data privacy-preserving model training across dispersed devices. Unfortunately, several challenges need to be concerned, such as the non-ID features of cross-client data and the possibility of backdoor attacks, which can result in inconsistent models, inefficiency, and security issues. To address these issues, we propose hypernetwork-based personalization and poisoning-free federated learning (HBIpFL), a unique architecture to improve security and personalization in FL. HBIpFL dramatically reduces communication overhead without sacrificing performance by using a hypernetwork-based parameter classifier to dynamically analyze and only upload the most important model parameters. Furthermore, it utilizes Local Gradient Ascent (LGA) methods to track training loss trends and identify possible backdoor intrusions, guaranteeing the resilience and dependability of the global model. We then compare HBIpFL to the state-of-the-art methods in the context of accuracy, communication efficiency, and defense against adversarial attacks. The results demonstrate that our HBIpFL offers a secure and effective FL environment for practical scenarios with a wide range of data distributions and strict privacy specifications.

Video inpainting is an important method to improve video quality and viewing experience. Traditional video inpainting methods require manual design, which has high operation costs and low efficiency. The deep learning method based on implicit neural representation provides an efficient automatic solution for video inpainting. It can improve the accuracy and effect of video inpainting by automatically learning the characteristics of the video and inpainting the video according to these characteristics. However, the scale of the embedding (feature) of the encoded video by the encoder is often small (16×2×4). As the input of the decoder, the embedding can only provide a small amount of feature information for the decoder, which greatly restricts the ability of video neural representation and further affects the effect of video inpainting. In this work, we construct a novel method of implicit neural representation for video inpainting, named I-NeRV, which enables a large-scale embedding (16×8×16) to enrich the information of the embedding. Meanwhile, the random mask mechanism is integrated into the coding part of neural representation, further improving the network feature extraction ability. Comprehensive experiments have proved that the video neural network in this paper has achieved significant optimization results in video inpainting tasks. On the representative dataset, video inpainting is more effective than SOTA, with a 3.47 PSNR improvement in inpainting metrics.

Decentralized storage platforms based on consortium blockchains have emerged in the spotlight of research and industry communities because they are flexible, transparent, and eliminated trust in contrast to the traditional centralized data-sharing model. However, due to wide attacking surfaces in a blockchain network, this decentralized data-sharing paradigm is subject to malicious data breaches. Untrusted blockchain nodes can directly obtain sensitive information from the query processing and their local storage. Several studies have been made for solving this dilemma, but they only focus on single-user settings and cannot be directly applied to multi-owners blockchain-based data sharing scenarios. In this paper, we introduce FedEDB, a federated and encrypted data store by using consortium blockchains. Unlike existing solutions that focus on single-user settings, our proposed schemes can efficiently support privacy-preserving and reliable multi-owner queries in the decentralized setting. We start from the practical key aggregation technique to construct the multi-owner search schemes and further refine the underling building blocks to enhance the security. Besides, we integrate the smart contract with our tailored zero-knowledge proof to enforce secure and reliable result verification protocol with fairness. We implement a prototype and thorough security analysis and comprehensive evaluation results confirm the practicability of our design.

Decentralized storage is recognized as a promising blockchain application for building trustworthy data-sharing services. Recently, privacy-preserving decentralized storage has attracted tremendous attention from the research community, due to the inherent transparent access properties of blockchain. That is, multi-users can search over encrypted data from multi-owners via untrusted storage servers. Private Set Intersection (PSI) is regarded as an ideal cryptographic scheme for this scenario because it allows multiple parties to collaboratively execute private set computations without revealing additional information. However, existing solutions do not consider the necessary verification and fault tolerance of PSI results, which is the indispensable security requirements in decentralized storage. To fill the gap, in this work, we introduce the first reliable PSI scheme for decentralized storage that provides results verifiability and fault tolerance for private set operations. Our design leverages authenticated indexing structures and Shamir's secret sharing algorithm for constructing our reliable PSI scheme with fault tolerance. To address the potential malicious behaviors of dishonest users or servers, we also propose a blockchain-assisted arbitration protocol that enables public arbitration in a privacy-preserving manner. We rigorously provide security analysis and complete the prototype implementation on Fabric. Extensive results demonstrate its practicability and feasibility for existing decentralized storage.