Understanding 3D scenes in mixed reality (MR) is crucial for advancing human-computer interaction, especially in MR applications that demand spatial awareness and contextual reasoning. While Vision-Language Models (VLMs) perform well in 2D image interpretation, they struggle to incorporate spatial context from 3D settings, which limits their effectiveness in MR scenarios. To address this issue, we introduce SAMR, a Spatial-Augmented Mixed Reality method designed to enhance VLMs for 3D scene understanding. Our system consists of three key modules. The first module, a spatial-segmented fusion module, uses FastSAM-based segmentation to create objectlevel meshes from head-mounted display (HMD) images. It maps extracted feature points to 3D coordinates through ray casting on the HMD-captured mesh and applies triangular facet fitting. The second module, a multimodal interaction module, combines gestures, gaze, and voice commands to enable intuitive interaction with 3D meshes for annotating prompts. The third module, a VLM integration module, processes data by merging annotated 2D images with user queries to form standardized prompts for the VLM. The VLM then generates responses linked to user-specified object meshes. By enhancing VLMs with spatial context and multimodal capabilities, SAMR greatly improves 3D scene interpretation. We demonstrate SAMR's effectiveness across six key application scenarios: object identification, relationship analysis, distance estimation, targeted object questioning, and cognitive assistance. This approach provides a robust framework for MR applications with AI agents.

Drawing freehand sketches of mechanical components on multimedia devices for AI-based engineering modeling has become a new trend. However, its development is being impeded because existing works cannot produce suitable sketches for data-driven research. These works either generate sketches lacking a freehand style or utilize generative models not originally designed for this task resulting in poor effectiveness. To address this issue, we design a two-stage generative framework mimicking the human sketching behavior pattern, called MSFormer, which is the first time to produce humanoid freehand sketches tailored for mechanical components. The first stage employs Open CASCADE technology to obtain multi-view contour sketches from mechanical components, filtering perturbing signals for the ensuing generation process. Meanwhile, we design a view selector to simulate viewpoint selection tasks during human sketching for picking out information-rich sketches. The second stage translates contour sketches into freehand sketches by a transformer-based generator. To retain essential modeling features as much as possible and rationalize stroke distribution, we introduce a novel edge-constraint stroke initialization. Furthermore, we utilize a CLIP vision encoder and a new loss function incorporating the Hausdorff distance to enhance the generalizability and robustness of the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance for generating freehand sketches in the mechanical domain. Project page: https://mcfreeskegen.github.io/.

Augmented reality (AR) has been widely employed in assembly guidance and maintenance as an excellent visualization tool. On this basis, AR technology combined with visual inspection has become a research topic to realize rapid and intuitive quality inspection while reducing operators' workload. This article proposes a novel multitarget, AR-based, assembly result inspection method, in which detected information is matched with prior knowledge via high-precision registration. First, a multimarker-based global registration method is designed to significantly improve the average registration accuracy in large scenes based on the mutual calibration of a few markers. Second, based on the correlation between the inspection target and its AR twin, an image containing multiple mechanical components is segmented according to the locations, and the local images are matched with the prior knowledge for evaluation. Finally, the inspection method is deployed to AR-based assembly inspection system, and its validity is verified on a rocket cabin imitation platform. Experiments show that the proposed inspection method can accurately segment the multitarget image into several images containing a single target according to the prior locations and can verify the assembly results of the targets, running at 15.1 fps.

Grasping an object when it is in an ungraspable pose is a challenging task, such as books or other large flat objects placed horizontally on a table. Inspired by human manipulation, we address this problem by pushing the object to the edge of the table and then grasping it from the hanging part. In this paper, we develop a model-free Deep Reinforcement Learning framework to synergize pushing and grasping actions. We first pre-train a Variational Autoencoder to extract high-dimensional features of input scenario images. One Proximal Policy Optimization algorithm with the common reward and sharing layers of Actor-Critic is employed to learn both pushing and grasping actions with high data efficiency. Experiments show that our one network policy can converge 2.5 times faster than the policy using two parallel networks. Moreover, the experiments on unseen objects show that our policy can generalize to the challenging case of objects with curved surfaces and off-center irregularly shaped objects. Lastly, our policy can be transferred to a real robot without fine-tuning by using CycleGAN for domain adaption and outperforms the push-to-wall baseline.

Contextual policy search methods have demonstrated the potential to acquire robotic skill generalization on trajectory-shaping-based tasks. However, it is still challenging for robotic contact-rich manipulation tasks because contact force regulation, reference trajectory adaptation, and task generalization must be fulfilled simultaneously. To this end, a hierarchical compliance-based contextual policy search (HC-CPS) approach is proposed to learn the robotic compliant skills for force, motion, and task adaptation. Specifically, the parameterized impedance-conditioned action space is proposed for reinforcement learning lower-level policy to obtain the compliance for reference motion regulation and contact force control, while a linear Gaussian contextual policy is formulated as the higher-level policy to optimize the context-conditioned impedance parameters for task generalization; therefore, a family of contact-rich manipulation tasks with multiple objectives is achieved. Moreover, data efficiency is further improved by two aspects: first, a variation encoder-decoder model is proposed to estimate the underlying constraints of impedance parameters over the actions, leading to the mitigated extrapolation error for lower-level policy off-policy learning; second, a composite forward model is proposed to generate artificial trajectories and reduce the reward bias for higher-level contextual policy learning. The HC-CPS approach is validated by three simulated manipulation tasks and the real-world dual peg-in-hole assembly tasks with two kinds of objectives, and the results demonstrate the effectiveness of HC-CPS.

Cover Caption: The cover image is based on the Research Article A deep transfer-learning-based dynamic reinforcement learning for intelligent tightening system by Wentao Luo et al., https://doi.org/10.1002/int.22345.

Reinforcement learning (RL) has been widely applied in the static environment with standard reward functions. For intelligent tightening tasks, it is a challenge to transform expert knowledge into a recognizable mathematical expression for RL agents. Changing assembly standards make the model repeat learning updated knowledge with a high time-cost. In addition, as the difficulty and low accuracy of designing reward functions, the RL model itself also limits its application in the complex and dynamic engineering environment. To solve the above problems, a deep transfer-learning-based dynamic reinforcement learning (DRL-DTL) is presented and applied in the intelligent tightening system. Specifically, a deep convolution transfer-learning model (DCTL) is presented to build a mathematical mapping between agents of the model and subjective knowledge, which endows agents to learn from human knowledge efficiently. Then, a dynamic expert library is established to improve the adaptability of algorithm to the changing environment. And an inverse RL based on prior knowledge is presented to acquire reward functions. Experiments are conducted on a tightening assembly system and the results show that the tightening robot with the proposed model can inspect quality problems during the tightening process autonomously and make an adjustment decision based on the optimal policy that the agent calculates.

Augmented assembly (AA) is being gradually incorporated into manual assembly owing to its abundant instruction forms, high efficiency of information transmission, and robust interactivity. However, a lack of a unified description model for assembly information and irregular instruction design forms leads to difficulties in instruction authoring and generation, inconveniences for data management, and low system portability. To address these issues, this paper presents an information-integration and instruction-authoring method for AA systems. First, design guidelines for the AA instructions are established with the aim of minimizing the user's cognitive load. After that, the designing of visual elements is performed based on the design guidelines, and the information model of the assembly information and instructions is built using Unified Modeling Language. Based on Extensible Markup Language files and Unity3D engine, a standard and rapid development process is presented for deploying the AA system to Hololens2 devices. The usability and replicability of the designed system are demonstrated, and a user study is carried out. The user's task performance is evaluated using completion time and errors as metrics. The NASA-task load index scale is used to estimate the user's workload, and the Likert scale reflects the user's subjective perception of the system and instructions. The results show that the proposed AA instructions can effectively improve the user's task performance; reduce their workload, especially the cognitive load; and improve their assembly experience.

Computation-Independent Model (CIM) to capture domain requirements and the transformation from CIM to the Platform-Independent Model (PIM) are two crucial parts of the Model-Driven Architecture (MDA). This paper presents an ontology-activity-based CIM modeling approach to achieve a semi-automatic transformation from CIM to PIM. It proposes that the key elements in business process modeling are activities and these should therefore form the basis in constructing the domain ontology. Aiming to provide the key description capability for the process model, it discusses the hierarchy of the model by adding an activity dimension between the object and process tiers. It also proposes a model-relevance-calculation-based method for extracting ontology activities from the process meta-models. Based on the presented model acquisition method, a decomposition approach is proposed to simplify the complexity of the transformation relationships between the CIM and PIM by introducing the concepts of ontology activities. A general framework surrounding the transformation from CIM to PIM is discussed. It uses the Web Ontology Language (OWL) to describe the ontology activity and considers the Unified Modeling Language (UML) to be the PIM.