Human beings are social animals. How to equip 3D autonomous characters with similar social intelligence that can perceive, understand and interact with humans remains an open yet foundamental problem. In this paper, we introduce SOLAMI, the first end-to-end Social vision-Language-Action (VLA) Modeling framework for Immersive interaction with 3D autonomous characters. Specifically, SOLAMI builds 3D autonomous characters from three aspects: 1) Social VLA Architecture: We propose a unified social VLA framework to generate multi-modal response (speech and motion) based on the user’s multimodal input to drive the character for social interaction. 2) Interactive Multimodal Data: We present Syn-MSI, a synthetic multimodal social interaction dataset generated by an automatic pipeline using only existing motion datasets to address the issue of data scarcity. 3) Immersive VR Interface: We develop a VR interface that enables users to immersively interact with these characters driven by various architectures. Extensive quantitative experiments and user study demonstrate that our framework leads to more precise and natural character responses (in both speech and motion) that align with user expectations with lower latency.

Control barrier certificate is an ingenious and practical approach of safety controller synthesis for cyber-physical systems. In this article, we present an approach for synthesizing safety controllers for controlled discrete-time systems subject to safety constraints. We first introduce a new type of$k$-inductive control barrier certificates ($k$-ICBCs), which relaxes the strict nonincreasing condition of general control barrier certificates. Apart from this, we propose a certificate synthesis framework that includes a learner and a verifier. They collaborate continuously to search for safety controllers and their corresponding$k$-ICBCs simultaneously. The learner obtains neural controllers and candidate$k$-ICBCs through supervised learning, while the verifier addresses a series of mixed integer linear programming problems to validate the candidate$k$-ICBCs or provide counterexamples to guide the learner further. Thanks to the less conservatism of$k$-inductive conditions, safety neural controllers, and$k$-ICBCs can be easily and quickly obtained. We showcase through benchmark examples that our method is efficient, and$k$-inductive conditions can improve the effectiveness of control barrier certificate synthesis methods by successfully verifying systems that are challenging to handle with general control barrier certificate conditions.

In-betweening is a technique for generating transitions given start and target character states. The majority of existing works require multiple (often 10) frames as input, which are not always available. In addition, they produce results that lack diversity, which may not fulfill artists' requirements. Addressing these gaps, our work deals with a focused yet challenging problem: generating diverse and high-quality transitions given exactly two frames (only the start and target frames). To cope with this challenging scenario, we propose a bi-directional motion generation and stitching scheme which generates forward and backward transitions from the start and target frames with two adversarial autoregressive networks, respectively, and stitches them midway between the start and target frames. In contrast to stitching at the start or target frames, where the ground truth cannot be altered, there is no strict midway ground truth. Thus, our method can capitalize on this flexibility and generate high-quality and diverse transitions simultaneously. Specifically, we employ conditional variational autoencoders (CVAEs) to implement our autoregressive networks and propose a novel stitching loss to stitch the bi-directional generated motions around the midway point. Extensive experiments demonstrate that our method achieves higher motion quality and more diverse results than existing methods on the LaFAN1, Human3.6m and AMASS datasets.

In this work, we present Digital Life Project, a framework utilizing language as the universal medium to build autonomous 3D characters, who are capable of engaging in social interactions and expressing with articulated body motions, thereby simulating life in a digital environment. Our framework comprises two primary components: 1) SocioMind: a meticulously crafted digital brain that models personalities with systematic few-shot exemplars, incorporates a reflection process based on psychology principles, and emulates autonomy by initiating dialogue topics; 2) MoMat-MoGen: a text-driven motion synthesis paradigm for controlling the character's digital body. It integrates motion matching, a proven industry technique to ensure motion quality, with cutting-edge advancements in motion generation for diversity. Extensive experiments demonstrate that each module achieves state-of-the-art performance in its respective domain. Collectively, they enable virtual characters to initiate and sustain dialogues autonomously, while evolving their socio-psychological states. Concurrently, these characters can perform contextually relevant bodily movements. Additionally, an extension of DLP enables a virtual character to recognize and appropriately respond to human players' actions.

Formal synthesis of safe controllers is essential for safety-critical cyber-physical systems. In this paper, we propose a novel counterexample guided approach for synthesizing safe controllers of nonlinear systems using Bayesian optimization enhanced reinforcement learning, to improve the efficiency of the training process while ensuring safety property. First, we utilize the control barrier function technique to establish a constrained Markov decision process, which enables us to learn an initial controller with minimal safety violations. We then design a counterexample guided policy refinement using Bayesian optimization, to fine-tune the initial controller based on the failure trajectories. Finally, we suggest a compensatory mechanism to correct the tuned controller to guarantee the safety property. We implement the CEGRLPR tool and evaluate its performance over a set of benchmarks. The experimental results demonstrate the effectiveness and efficiency of our approach.

The technological advancements in digital twins (DTs) and next-generation networks boost the development of the Industrial Internet of Things (IIoT). However, the increasing concern for data privacy hinders the further development of IIoT. By manipulating its own dataset and training machine learning models locally, federated learning (FL) has been regarded as a promising technology to establish DT models for IIoT. Nevertheless, the vast amount of model parameters, limited computation and communication resources, and the dynamic industrial environments pose great challenges for FL. In this paper, we present a novel FL framework integrating with DTs in the resource-constrained and dynamic IIoT environment. The clustering algorithm is exploited to design an efficient client selection approach. Furthermore, we propose an efficient model aggregation approach to improve the training efficiency. A resource allocation problem is then formulated and solved based on deep reinforcement learning. Numerical results show that the proposed client selection and resource allocation approach reduces the time cost compared with the baseline schemes.