Understanding dynamic 4D scenes from an egocentric perspective—modeling changes in 3D spatial structure over time—is crucial for human–machine interaction, autonomous navigation, and embodied intelligence. While existing egocentric datasets contain dynamic scenes, they lack unified 4D annotations and task-driven evaluation protocols for fine-grained spatio-temporal reasoning, especially on motion of objects and human, together with their interactions. To address this gap, we introduce EgoDynamic4D, a novel QA benchmark on highly dynamic scenes, comprising RGB-D video, camera poses, globally unique instance masks, and 4D bounding boxes. We construct 927K QA pairs accompanied by explicit Chain-of-Thought (CoT), enabling verifiable, step-by-step spatio-temporal reasoning. We design 12 dynamic QA tasks covering agent motion, human–object interaction, trajectory prediction, relation understanding, and temporal–causal reasoning, with fine-grained, multidimensional metrics. To tackle these tasks, we propose an end-to-end spatio-temporal reasoning framework that unifies dynamic and static scene information, using instance-aware feature encoding, time and camera encoding, and spatially adaptive down-sampling to compress large 4D scenes into token sequences manageable by LLMs. Experiments on EgoDynamic4D show that our method consistently outperforms baselines, validating the effectiveness of multimodal temporal modeling for egocentric dynamic scene understanding.

Recent advances in large reasoning models (LRMs) have significantly enhanced long-chain reasoning capabilities over standard large language models (LLMs). However, LRMs often produce unnecessarily lengthy outputs even for simple queries, leading to inefficiencies or even accuracy degradation compared to LLMs. To address this, we propose CP-Router, a training-free, model-agnostic routing framework that dynamically selects between an LLM and an LRM, demonstrated with multiple-choice question answering (MCQA) prompts. The routing decision is guided by the prediction uncertainty estimates derived via Conformal Prediction (CP), which provides rigorous coverage guarantees. To improve uncertainty differentiation across inputs, we introduce Full and Binary Entropy (FBE), a novel entropy-based criterion that adaptively selects the appropriate CP threshold. Experiments across MCQA and QA benchmarks—including mathematics, logical reasoning, and Chinese chemistry—demonstrate that CP-Router efficiently reduces token usage while maintaining or even improving accuracy compared to using LRM alone. We further demonstrate the generality and robustness of CP-Router by extending it to diverse model pairings beyond the LLM–LRM setting.

Modeling 3D language fields with Gaussian Splatting for open-ended language queries has recently garnered increasing attention. However, recent 3DGS-based models leverage view-dependent 2D foundation models to refine 3D semantics but lack a unified 3D representation, leading to view inconsistencies. Additionally, inherent open-vocabulary challenges cause inconsistencies in object and relational descriptions, impeding hierarchical semantic understanding. In this paper, we propose Hi-LSplat, a view-consistent Hierarchical Language Gaussian Splatting work for 3D open-vocabulary querying. To achieve view-consistent 3D hierarchical semantics, we first lift 2D features to 3D features by constructing a 3D hierarchical semantic tree with layered instance clustering, which addresses the view inconsistency issue caused by 2D semantic features. Besides, we introduce instance-wise and part-wise contrastive losses to capture all-sided hierarchical semantic representations. Notably, we construct two hierarchical semantic datasets to better assess the model's ability to distinguish different semantic levels. Extensive experiments highlight our method's superiority in 3D open-vocabulary segmentation and localization. Its strong performance on hierarchical semantic datasets underscores its ability to capture complex hierarchical semantics within 3D scenes.

Spiking Neural Networks (SNNs) are emerging as a promising alternative to Artificial Neural Networks (ANNs) due to their inherent energy efficiency. Owing to the inherent sparsity in spike generation within SNNs, the in-depth analysis and optimization of intermediate output spikes are often neglected. This oversight significantly restricts the inherent energy efficiency of SNNs and diminishes their advantages in spatiotemporal feature extraction, resulting in a lack of accuracy and unnecessary energy expenditure. In this work, we analyze the inherent spiking characteristics of SNNs from both temporal and spatial perspectives. In terms of spatial analysis, we find that shallow layers tend to focus on learning vertical variations, while deeper layers gradually learn horizontal variations of features. Regarding temporal analysis, we observe that there is not a significant difference in feature learning across different time steps. This suggests that increasing the time steps has limited effect on feature learning. Based on the insights derived from these analyses, we propose a Frequency-based Spatial-Temporal Attention (FSTA) module to enhance feature learning in SNNs. This module aims to improve the feature learning capabilities by suppressing redundant spike features. The experimental results indicate that the introduction of the FSTA module significantly reduces the spike firing rate of SNNs, demonstrating superior performance compared to state-of-the-art baselines across multiple datasets.

Recent advancements in large language models (LLMs) have given rise to the LLM-as-a-judge paradigm, showcasing their potential to deliver human-like judgments. However, in the field of machine translation (MT) evaluation, current LLM-as-a-judge methods fall short of learned automatic metrics. In this paper, we propose Multidimensional Multi-Agent Debate (M-MAD), a systematic LLM-based multi-agent framework for advanced LLM-as-a-judge MT evaluation. Our findings demonstrate that M-MAD achieves significant advancements by (1) decoupling heuristic MQM criteria into distinct evaluation dimensions for fine-grained assessments; (2) employing multi-agent debates to harness the collaborative reasoning capabilities of LLMs; (3) synthesizing dimension-specific results into a final evaluation judgment to ensure robust and reliable outcomes. Comprehensive experiments show that M-MAD not only outperforms all existing LLM-as-a-judge methods but also competes with state-of-the-art reference-based automatic metrics, even when powered by a suboptimal model like GPT-4o mini. Detailed ablations and analysis highlight the superiority of our framework design, offering a fresh perspective for LLM-as-a-judge paradigm. Our code and data are publicly available at https://github.com/SU-JIAYUAN/M-MAD.

The strong capability of large language models (LLMs) has been applied to information extraction (IE) through either retrieval augmented prompting or instruction tuning (IT). However, the best way to incorporate information with LLMs for IE remains an open question. In this paper, we explore Retrieval Augmented Instruction Tuning (RA-IT) for IE, focusing on the task of open named entity recognition (NER). Specifically, for each training sample, we retrieve semantically similar examples from the training dataset as the context and prepend them to the input of the original instruction. To evaluate our RA-IT approach more thoroughly, we construct a Chinese IT dataset for open NER and evaluate RA-IT in both English and Chinese scenarios. Experimental results verify the effectiveness of RA-IT across various data sizes and in both English and Chinese scenarios. We also conduct thorough studies to explore the impacts of various retrieval strategies in the proposed RA-IT framework.

Recent large language models have become popular for achieving state-of-the-art performance in various natural language processing tasks, especially in zero-shot applications where fine-tuning is not required. However, these models underperform in text classification compared to fine-tuned models, due to limitations in reasoning ability and token constraints in in-context learning. Although extensive research explores large language models for text classification, few studies address multi-class classification with these models. This study introduces a new multi-class classification framework using a hierarchical label filtering strategy to manage long prompts in text classification. The influence of label sequence on classification accuracy is further investigated, demonstrating that optimized label arrangements significantly boost performance. Additionally, the study compares the performance of human-defined and model-generated labels in text classification, and analyze performance disparities across models for the same classification task.

Although deep learning-based intelligent fault diagnosis methods have garnered significant research interest, their impressive performance relies on the assumption that the training and test data share the same distribution. This constraint limits their effectiveness in scenarios where these distributions differ. Unsupervised domain adaptation techniques for fault diagnosis address this issue; however, most existing approaches focus solely on data from a single source domain, which overlooks the potential for leveraging multiple source domains to enhance diagnostic accuracy. In this paper, we propose a multi-source domain adaptation fault diagnosis method using enhanced multi-kernel maximum mean discrepancy to effectively extract features from multiple source domains for accurate target domain fault diagnosis. Extensive experiments on two fault diagnosis datasets demonstrate the superior performance of the proposed method.

Spiking Neural Networks (SNNs), inspired by the human brain, offer significant computational efficiency through discrete spike-based information transfer. Despite their potential to reduce inference energy consumption, a performance gap persists between SNNs and Artificial Neural Networks (ANNs), primarily due to current training methods and inherent model limitations. While recent research has aimed to enhance SNN learning by employing knowledge distillation (KD) from ANN teacher networks, traditional distillation techniques often overlook the distinctive spatiotemporal properties of SNNs, thus failing to fully leverage their advantages. To overcome these challenge, we propose a novel logit distillation method characterized by temporal separation and entropy regularization. This approach improves existing SNN distillation techniques by performing distillation learning on logits across different time steps, rather than merely on aggregated output features. Furthermore, the integration of entropy regularization stabilizes model optimization and further boosts the performance. Extensive experimental results indicate that our method surpasses prior SNN distillation strategies, whether based on logit distillation, feature distillation, or a combination of both. Our project is available at https://github.com/yukairong/TSER.

Self-supervised learning (SSL) models are vulnerable to backdoor attacks. Existing backdoor attacks that are effective in SSL often involve noticeable triggers, like colored patches or visible noise, which are vulnerable to human inspection. This paper proposes an imperceptible and effective backdoor attack against self-supervised models. We first find that existing imperceptible triggers designed for supervised learning are less effective in compromising self-supervised models. We then identify this ineffectiveness is attributed to the overlap in distributions between the backdoor and augmented samples used in SSL. Building on this insight, we design an attack using optimized triggers disentangled with the augmented transformation in the SSL, while remaining imperceptible to human vision. Experiments on five datasets and six SSL algorithms demonstrate our attack is highly effective and stealthy. It also has strong resistance to existing backdoor defenses. Our code can be found at https://github.com/Zhang-Henry/INACTIVE.

Spiking Neural Networks (SNNs) have gained significant attention due to their biological plausibility and energy efficiency, making them promising alternatives to Artificial Neural Networks (ANNs). However, the performance gap between SNNs and ANNs remains a substantial challenge hindering the widespread adoption of SNNs. In this paper, we propose a Spatial-Temporal Attention Aggregator SNN (STAA-SNN) framework, which dynamically focuses on and captures both spatial and temporal dependencies. First, we introduce a spike-driven self-attention mechanism specifically designed for SNNs. Additionally, we pioneeringly incorporate position encoding to integrate latent temporal relationships into the incoming features. For spatial-temporal information aggregation, we employ step attention to selectively amplify relevant features to variant steps. Finally, we implement a time-step random dropout strategy to avoid local optima. The framework demonstrates exceptional performance across diverse datasets and exhibits strong generalization capabilities. Notably, STAA-SNN achieves state-of-the-art results on neuromorphic datasets CIFAR10-DVS of 82.10% and with performances of 97.14%, 82.05% and 70.40% on the static datasets CIFAR-10, CIFAR-100 and ImageNet, respectively. Furthermore, this model exhibits improved performance ranging from 0.33% to 2.80% with fewer time steps.

Semantic information is crucial for human awareness. The ability to extract such information interactively from brain activity using non-invasive technologies like functional Magnetic Resonance Imaging (fMRI) is valuable for medical assistive technologies. However, research in this domain remains relatively limited. To address this gap, we proposes BrainChat, an interactive framework designed to decode semantic information from fMRI. BrainChat leverages a large-scale vision-language model, functioning through fMRI-based captioning and, optionally, question answering. First, a pair of fMRI encoder and decoder is trained to map fMRI data into a latent space representation using Masked Brain Modeling, a self-supervised approach. On the second stage, a projector is added to align these fMRI representations with both pretrained image and text embeddings, yielding a unified representation. A text decoder is also added at this stage, adopting cross-attention with the unified fMRI representation to guide the generation of semantic information. During this stage, the fMRI encoder, the projector, and the text decoder are trained together by minimizing a combined contrastive loss and caption loss. BrainChat achieves state-of-the-art performance in fMRI captioning and implements fMRI question answering, enabling interactive clinical applications. The code is available on Github1.

Spiking Neural Networks (SNNs) are valued for their ability to process spatio-temporal information efficiently, offering biological plausibility, low energy consumption, and compatibility with neuromorphic hardware. However, the commonly used Leaky Integrate-and-Fire (LIF) model overlooks neuron heterogeneity and independently processes spatial and temporal information, limiting the expressive power of SNNs. In this paper, we propose the Dual Adaptive Leaky Integrate- and-Fire (DA-LIF) model, which introduces spatial and temporal tuning with independently learnable decays. Evaluations on both static (CIFAR10/100, ImageNet) and neuromorphic datasets (CIFAR10-DVS, DVS128 Gesture) demonstrate superior accuracy with fewer timesteps compared to state-of-the-art methods. Importantly, DA-LIF achieves these improvements with minimal additional parameters, maintaining low energy consumption. Extensive ablation studies further highlight the robustness and effectiveness of the DA-LIF model.

Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30\%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.

Spiking Neural Networks (SNNs) are increasingly recognized for their biological plausibility and energy efficiency, positioning them as strong alternatives to Artificial Neural Networks (ANNs) in neuromorphic computing applications. SNNs inherently process temporal information by leveraging the precise timing of spikes, but balancing temporal feature utilization with low energy consumption remains a challenge. In this work, we introduce Temporal Shift module for Spiking Neural Networks (TS-SNN), which incorporates a novel Temporal Shift (TS) module to integrate past, present, and future spike features within a single timestep via a simple yet effective shift operation. A residual combination method prevents information loss by integrating shifted and original features. The TS module is lightweight, requiring only one additional learnable parameter, and can be seamlessly integrated into existing architectures with minimal additional computational cost. TS-SNN achieves state-of-the-art performance on benchmarks like CIFAR-10 (96.72%), CIFAR-100 (80.28%), and ImageNet (70.61%) with fewer timesteps, while maintaining low energy consumption. This work marks a significant step forward in developing efficient and accurate SNN architectures.

Healthcare data frequently contain a substantial proportion of missing values, necessitating effective time series imputation to support downstream disease diagnosis tasks. However, existing imputation methods focus on discrete data points and are unable to effectively model sparse data, resulting in particularly poor performance for imputing substantial missing values. In this paper, we propose a novel approach, ImputeINR, for time series imputation by employing implicit neural representations (INR) to learn continuous functions for time series. ImputeINR leverages the merits of INR in that the continuous functions are not coupled to sampling frequency and have infinite sampling frequency, allowing ImputeINR to generate fine-grained imputations even on extremely sparse observed values. Extensive experiments conducted on eight datasets with five ratios of masked values show the superior imputation performance of ImputeINR, especially for high missing ratios in time series data. We also validate that applying ImputeINR to impute missing values in healthcare data enhances the performance of downstream disease diagnosis tasks.

Existing fMRI-to-image reconstruction methods have made progress in decoding visual images from fMRI but continue to struggle with capturing finer details like shape and color. Inspired by the human brain’s processing of visual information—particularly its encoding at multiple spatial scales across various cortical areas, each with distinct sensitivities to different stimuli, a phenomenon we term Multiscale Encoding and Perceptual Specificity—we propose a method to decode human vision from fMRI data. Specifically, we use a multiscale module to integrate information across different scales, and a channel attention module to focus on features related to the reconstruction task. Notably, our research reveals distinct feature sensitivities in the multiscale module and channel attention module to fMRI data, indicating a potential functional decoupling, as confirmed through experiments. This further guides architectural refinements, as we reorganized the network modules, which led to an improved reconstruction performance. Furthermore, our analysis reveals a correlation between visual stimulus size, fMRI response scale, and various attention scales within the multiscale module. The experimental results help us understand cognitive psychology and neuroscience from the perspective of neural networks. The code is available on GitHub.

Spiking Neural Networks (SNNs) have emerged as a promising approach for energy-efficient and biologically plausible computation. However, due to limitations in existing training methods and inherent model constraints, SNNs often exhibit a performance gap when compared to Artificial Neural Networks (ANNs). Knowledge distillation (KD) has been explored as a technique to transfer knowledge from ANN teacher models to SNN student models to mitigate this gap. Traditional KD methods typically use Kullback-Leibler (KL) divergence to align output distributions. However, conventional KL-based approaches fail to fully exploit the unique characteristics of SNNs, as they tend to overemphasize high-probability predictions while neglecting low-probability ones, leading to suboptimal generalization. To address this, we propose Head-Tail Aware Kullback-Leibler (HTA-KL) divergence, a novel KD method for SNNs. HTA-KL introduces a cumulative probability-based mask to dynamically distinguish between high- and low-probability regions. It assigns adaptive weights to ensure balanced knowledge transfer, enhancing the overall performance. By integrating forward KL (FKL) and reverse KL (RKL) divergence, our method effectively align both head and tail regions of the distribution. We evaluate our methods on CIFAR-10, CIFAR-100 and Tiny ImageNet datasets. Our method outperforms existing methods on most datasets with fewer timesteps.

Time series anomaly detection aims to identify unusual patterns in data or deviations from systems' expected behavior. The reconstruction-based methods are the mainstream in this task, which learn point-wise representation via unsupervised learning. However, the unlabeled anomaly points in training data may cause these reconstruction-based methods to learn and reconstruct anomalous data, resulting in the challenge of capturing normal patterns. In this paper, we propose a time series anomaly detection method based on implicit neural representation (INR) reconstruction, named TSINR, to address this challenge. Due to the property of spectral bias, TSINR enables prioritizing low-frequency signals and exhibiting poorer performance on high-frequency abnormal data. Specifically, we adopt INR to parameterize time series data as a continuous function and employ a transformer-based architecture to predict the INR of given data. As a result, the proposed TSINR method achieves the advantage of capturing the temporal continuity and thus is more sensitive to discontinuous anomaly data. In addition, we further design a novel form of INR continuous function to learn inter- and intra-channel information, and leverage a pre-trained large language model to amplify the intense fluctuations in anomalies. Extensive experiments demonstrate that TSINR achieves superior overall performance on both univariate and multivariate time series anomaly detection benchmarks compared to other state-of-the-art reconstruction-based methods. Our codes are available here.

Nested NER tasks have some challenges in specific domains, such as biomedical and industrial fields, particularly due to low resource and class imbalance, which impede its wide application. In this study, we design a novel loss EIoU-EMC, by enhancing the implement of Intersection over Union loss and Multi-class loss. Our proposed method specially leverages the information of entity boundary and entity classification, thereby enhancing the model's capacity to learn from a limited number of data samples. To validate the performance of this innovative method in enhancing NER task, we conducted experiments on three distinct biomedical NER datasets and one dataset constructed by ourselves from industrial complex equipment maintenance documents. Comparing to strong baselines, our method demonstrates the competitive performance across all datasets. During the experimental analysis, our proposed method exhibits significant advancements in entity boundary recognition and entity classification. Our code and data are available at https://github.com/luminous11/EIoU-EMC/

Surface defect is a crucial factor affecting the product quality of steel products. Current studies mainly focus on defect recognition and classification using machine vision-based algorithms, which lack the trace of potential causes and the reuse of experiential knowledge. To address this issue, we construct a knowledge graph for steel surface defects by fusing the multi-source and heterogeneous industrial data, including process parameters, chemical compositions, defect images, operation logs and empirical knowledge. A policy-based reinforcement learning approach is developed to solve the path reasoning problem over the industrial knowledge graph in defect detection and diagnosis. The approach employs two agents to explore the path efficiently from opposite directions, utilizes an integrated reward function that comprehensively considers the path direction, path length and entity distance to perform action selection, and adopts the path sharing mechanism and the prior knowledge to update selection policy. Experimental comparisons with the state-of-the-art knowledge reasoning algorithms on two benchmark datasets, NELL-995 and FB15K-237, validate the performance and merits of the proposed method. The effectiveness of the proposed method is also evaluated on a practical steel surface defect dataset, and the results show that our approach performs well in knowledge reasoning on the surface defect graph.Note to Practitioners—The surface quality of products has become a widely concerned focus in manufacturing industries. With the development of industrial IoT and Cyber-physical system technologies, more and more industrial data has been collected, and machine learning-based algorithms have been developed and applied to the recognition of detect defects. However, the algorithms do not take full advantage of the multi-source and heterogeneous defect-related data. On the other hand, it is also difficult to accumulate, inherit and reuse the experts’ knowledge of solving historical cases in the long-term production process. In order to deal with the above obstacles, we apply the knowledge graph for steel surface defect detection. In the proposed approach, a policy-based reinforcement learning algorithm is developed to solve the path reasoning problem over the industrial knowledge graph. To further improve the performance of our algorithm, we employ two agents to explore the path efficiently from opposite directions, utilize an integrated reward function which comprehensively considers the path direction, path length and entity distance to perform action selection, adopt the path sharing mechanism and updated selection policy to reuse the prior knowledge. As a result, our algorithm obtains high precision in knowledge reasoning tasks on two benchmark datasets and a practical steel surface defect dataset compared with some existing algorithms. Hence, it can be readily applied to real surface defect detection problems and facilitates intelligent manufacturing in steel production.

Class-incremental fault diagnosis requires a model to adapt to new fault classes while retaining previous knowledge. However, limited research exists for imbalanced and long-tailed data. Extracting discriminative features from few-shot fault data is challenging, and adding new fault classes often demands costly model retraining. Moreover, incremental training of existing methods risks catastrophic forgetting, and severe class imbalance can bias the model's decisions toward normal classes. To tackle these issues, we introduce a supervised contrastive knowledge distillation for class incremental fault diagnosis (SCLIFD) framework proposing supervised contrastive knowledge distillation for improved representation learning capability and less forgetting, a novel prioritized exemplar selection method for sample replay to alleviate catastrophic forgetting, and the random forest classifier to address the class imbalance. Extensive experimentation on simulated and real-world industrial datasets across various imbalance ratios demonstrates the superiority of SCLIFD over existing approaches.

Extending image-based Large Multimodal Models (LMMs) to video-based LMMs always requires temporal modeling in the pre-training. However, training the temporal modules gradually erases the knowledge of visual features learned from various image-text-based scenarios, leading to degradation in some downstream tasks. % Adapting pre-trained video-based large language models (LLMs) to downstream fine-grained video understanding tasks always requires modeling on temporal modules. However, training the temporal modules during video pretraining gradually erases the knowledge of visual features learned from various image-text-based scenarios, leading to degradation in some downstream tasks. % Instead of tuning video-based LLMs to downstream tasks, To address this issue, in this paper, we introduce a novel, efficient transfer approach termed MTransLLAMA, which employs transfer learning from pre-trained image LMMs for fine-grained video tasks with only small-scale training sets. Our method enablesfewer trainable parametersand achievesfaster adaptationandhigher accuracythan pre-training video-based LMM models. Specifically, our method adopts early fusion between textual and visual features to capture fine-grained information, reuses spatial attention weights in temporal attentions for cyclical spatial-temporal reasoning, and introduces dynamic attention routing to capture both global and local information in spatial-temporal attentions. Experiments demonstrate that across multiple datasets and tasks, without relying on video pre-training, our model achieves state-of-the-art performance, enabling lightweight and efficient transfer from image-based LMMs to fine-grained video tasks.

The gearbox in a wind turbine is an important component, whereas it often operates in harsh environments, resulting in a relatively high failure rate. Fault diagnosis of wind turbine gearboxes by means of vibration signals is a feasible solution, but in practice, the vibration signals collected by sensors are often accompanied by noise, which affects the accuracy of fault diagnosis. In this paper, a noise-robust convolutional neural network (NRCNN) is utilized for fault diagnosis of wind turbine gearboxes with noisy vibration signals. The structure of multilayer convolutional layers and fully connected layers provides the NRCNN with excellent feature extraction capability, while the introduction of the dropout layer enables the NRCNN to have better generalization capability. Comprehensive experiments on two gearbox vibration datasets demonstrate that the NRCNN could accurately perform the diagnosis of vibration signals with noise.

Medical generative models, acknowledged for their high-quality sample generation ability, have accelerated the fast growth of medical applications. However, recent works concentrate on separate medical generation models for dis-tinct medical tasks and are restricted to inadequate medi-cal multimodal knowledge, constraining medical compre-hensive diagnosis. In this paper, we propose MedM2G, a Medical Multi-Modal Generative framework, with the key innovation to align, extract, and generate medical multimodal within a unified model. Extending beyond single or two medical modalities, we efficiently align medical multimodal through the central alignment approach in the unified space. Significantly, our framework extracts valuable clini-cal knowledge by preserving the medical visual invariant of each imaging modal, thereby enhancing specific medical information for multimodal generation. By conditioning the adaptive cross-guided parameters into the multi-flow diffusion framework, our model promotes flexible interactions among medical multimodalfor generation. MedM2G is the first medical generative model that unifies medical generation tasks of text-to-image, image-to-text, and unified generation of medical modalities (CT, MRI, X-ray). It performs 5 medical generation tasks across 10 datasets, consistently outperforming various state-of-the-art works.

Retrieval-augmented language model (RALM) represents a significant advancement in mitigating factual hallucination by leveraging external knowledge sources.However, the reliability of the retrieved information is not always guaranteed, and the retrieval of irrelevant data can mislead the response generation.Moreover, standard RALMs frequently neglect their intrinsic knowledge due to the interference from retrieved information.In instances where the retrieved information is irrelevant, RALMs should ideally utilize their intrinsic knowledge or, in the absence of both intrinsic and retrieved knowledge, opt to respond with "unknown" to avoid hallucination.In this paper, we introduces CHAIN-OF-NOTE (CON), a novel approach to improve robustness of RALMs in facing noisy, irrelevant documents and in handling unknown scenarios.The core idea of CON is to generate sequential reading notes for each retrieved document, enabling a thorough evaluation of their relevance to the given question and integrating this information to formulate the final answer.Our experimental results show that GPT-4, when equipped with CON, outperforms the CHAIN-OF-THOUGHT approach.Besides, we utilized GPT-4 to create 10K CON data, subsequently trained on LLaMa-2 7B model.Our experiments across four open-domain QA benchmarks show that fine-tuned RALMs equipped with CON significantly outperform standard fine-tuned RALMs.

Large language models (LLMs) have demonstrated emergent capabilities across diverse reasoning tasks via popular Chains-of-Thought (COT) prompting.However, such a simple and fast COT approach often encounters limitations in dealing with complicated problems, while a thorough method, which considers multiple reasoning pathways and verifies each step carefully, results in slower inference.This paper addresses the challenge of enabling LLMs to autonomously select between fast and slow inference methods, thereby optimizing both efficiency and effectiveness.We introduce a dynamic decision-making framework that categorizes tasks into two distinct pathways: 'Fast', designated for tasks where the LLM quickly identifies a high-confidence solution, and 'Slow', allocated for tasks that the LLM perceives as complex and for which it has low confidence in immediate solutions as well as requiring more reasoning paths to verify.Experiments on five popular reasoning benchmarks demonstrated the superiority of the Dyna-Think over baselines.

Knowledge graph embedding (KGE) has caught significant interest for its effectiveness in knowledge graph completion (KGC), specifically link prediction (LP), with recent KGE models cracking the LP benchmarks. Despite the rapidly growing literature, insufficient attention was paid to the cooperation between humans and AI on KG. However, humans' capability to analyze graphs conceptually may further improve the efficacy of KGE models with semantic information. To this effect, we carefully designed a human-AI team (HAIT) system dubbed KG-HAIT, which harnesses the human insights on KG by leveraging fully human-designed ad-hoc dynamic programming (DP) on KG to produce human insightful feature (HIF) vectors that capture the subgraph structural feature and semantic similarities. By integrating HIF vectors into the training of KGE models, notable improvements are observed across various benchmarks and metrics, accompanied by accelerated model convergence. Our results underscore the effectiveness of human-designed DP in the task of LP, emphasizing the pivotal role of collaboration between humans and AI on KG. We open avenues for further exploration and innovation through KG-HAlT, paving the way towards more effective and insightful KG analysis techniques.

Although convolution neural network (CNN) has achieved certain success in fault detection and diagnosis (FDD) tasks in the chemical engineering industry, the performance and credibility of CNN-based FDD methods are greatly limited by two factors. First, CNN relies upon strong temporal/spatial correlation in data, which is very difficult to obtain in generic chemical tabular data. Second, most CNN methods have poor interpretability due to the encapsulation mechanism of feature extraction, and thus there is great difficulty in identifying the root-cause features from the results obtained using these methods. To address these difficulties, we propose an order-invariant and interpretable dilated convolution neural network (OIDLCNN), which is composed of feature clustering, dilated convolution, and a deep Shapley additive explanations (SHAP) explainer. Specifically, the feature clustering technique is adopted to reorder the features thus those with strong correlations are placed to be adjacent to each other. The large receptive field of dilated convolution can capture long-range correlations so then it can further recover the feature correlations and improve the classification performance. Last but not least, the proposed method provides interpretability by including the SHAP values to quantify feature contribution and identify the root-cause feature as the one with the highest contribution. Computational experiments are conducted on the Tennessee Eastman chemical process benchmark dataset. Compared with the other state-of-the-art methods, the proposed OIDLCNN-SHAP method achieves better performance in capturing feature correlations, detecting faults, and identifying the root-cause features.Note to Practitioners—Fault detection and diagnosis (FDD) is significant for reducing maintenance costs and improving safety in chemical processes. In this paper, we investigate the difficulty in detecting faults and identifying the root-cause features in chemical multivariate processes. This work was motivated by the fact that the existing CNN-based FDD algorithms are not designed for generic chemical tabular data and fail to capture vital information about feature correlations. In addition, the commonly used bayesian network-based root cause analysis methods are expensive since they require much prior knowledge and expert rules. We present a novel framework to capture the unique feature correlations and obtain the root-cause features without any expert knowledge. The proposed method provides an automatic and low-cost way for chemical process FDD tasks. It can be further integrated into the condition monitoring system of real chemical processes to analyze potential faults and identify the corresponding root causes in real time.

Fault diagnosis is crucial in monitoring machines within industrial processes. With the increasing complexity of working conditions and demand for safety during production, diverse diagnosis methods are required, and an integrated fault diagnosis system capable of handling multiple tasks is highly desired. However, the diagnosis subtasks are often studied separately, and the current methods still need improvement for such a generalized system. To address this issue, we propose the generalized out-of-distribution fault diagnosis (GOOFD) framework to integrate diagnosis subtasks. Additionally, a unified fault diagnosis method based on internal contrastive learning and Mahalanobis distance is put forward to underpin the proposed generalized framework. The method involves feature extraction through internal contrastive learning and outlier recognition based on the Mahalanobis distance. Our proposed method can be applied to multiple fault diagnosis tasks and achieve better performance than the existing single-task methods. Experiments are conducted on benchmark and practical process datasets, indicating the effectiveness of the proposed framework.

Medical vision-language pre-training (Med-VLP) models have recently accelerated the fast-growing medical diagnostics application. However, most Med-VLP models learn task-specific representations independently from scratch, thereby leading to great inflexibility when they work across multiple fine-tuning tasks. In this work, we proposeUniDCP, aUnified medical vision-language model withDynamicCross-modal learnablePrompts, which can be plastically applied to multiple medical vision-language tasks within a unified model. Specifically, we explicitly construct a unified framework to harmonize diverse inputs from multiple pre-training tasks by leveraging cross-modal prompts for unification, which accordingly can accommodate heterogeneous medical fine-tuning tasks within a same model. Furthermore, we conceive a dynamic cross-modal prompt optimizing strategy that optimizes the prompts within the shareable space for implicitly processing the shareable clinic knowledge. UniDCP is the first Med-VLP model capable of performing all 8 medical uni-modal and cross-modal tasks over 14 corresponding datasets, consistently yielding superior results over diverse state-of-the-art methods.

In real industrial processes, fault diagnosis methods are required to learn from limited fault samples since the procedures are mainly under normal conditions and the faults rarely occur. Although attention mechanisms have become increasingly popular for the task of fault diagnosis, the existing attention-based methods are still unsatisfying for the above practical applications. First, pure attention-based architectures like transformers need a substantial quantity of fault samples to offset the lack of inductive biases thus performing poorly under limited fault samples. Moreover, the poor fault classification dilemma further leads to the failure of the existing attention-based methods to identify the root causes. To develop a solution to the aforementioned problems, we innovatively propose a supervised contrastive convolutional attention mechanism (SCCAM) with ante-hoc interpretability, which solves the root cause analysis problem under limited fault samples for the first time. First, accurate classification results are obtained under limited fault samples. More specifically, we integrate the convolutional neural network (CNN) with attention mechanisms to provide strong intrinsic inductive biases of locality and spatial invariance, thereby strengthening the representational power under limited fault samples. In addition, we ulteriorly enhance the classification capability of the SCCAM method under limited fault samples by employing the supervised contrastive learning (SCL) loss. Second, a novel ante-hoc interpretable attention-based architecture is designed to directly obtain the root causes without expert knowledge. The convolutional block attention module (CBAM) is utilized to directly provide feature contributions behind each prediction thus achieving feature-level explanations. The proposed SCCAM method is testified on a continuous stirred tank heater (CSTH) and the Tennessee Eastman (TE) industrial process benchmark. Three common fault diagnosis scenarios are covered, including a balanced scenario for additional verification and two scenarios with limited fault samples (i.e., imbalanced scenario and long-tail scenario). The effectiveness of the presented SCCAM method is evidenced by the comprehensive results that show our method outperforms the state-of-the-art methods in terms of fault classification and root cause analysis.

Medical zero-shot relation triplet extraction, referred to as Med-ZeroRTE, requires the model to extract triplets comprising entities and relations from medical sentences. Importantly, the sentences include relations that were unseen during the model’s training phase. While Med-ZeroRTE had not been formally explored before this work, the limited availability of medical datasets, influenced by privacy concerns and annotation costs, emphasizes the necessity of exploring Med-ZeroRTE. This exploration faces two main challenges: Firstly, there is a gap of work specifically focused on triplet extraction from medical text in a zero-shot setting. Secondly, while a few approaches tackle the general zero-shot problems by employing generative models to produce synthetic data for unseen classes, the quality of some synthetic data remains suboptimal. Therefore, we propose a novel Enhanced Generator - Discriminator - Extractor framework (EGDE), which consists of three core modules, a prompt-tuned generator for generating synthetic samples given unseen relations, a fine-tuned discriminator for filtering qualified synthetic samples, a prompt-tuned extractor for extracting predicted medical triplets, to resolve Med-ZeroRTE and mitigate issues related to poor synthetic samples. The proposed framework is shown to be effective and superior compared to several robust baselines in experiments conducted on two distinct dataset settings.

Compared with traditional multivariate statistical techniques, deep neural networks have been frequently used for single-mode fault diagnosis and have shown promising results. However, in the real world, a complex industrial process may have several modes and fewer fault samples than normal. Although several multivariate statistical techniques focus on multimode fault diagnosis, those methods usually identify modes and then locally diagnose faults while ignoring relative information across modes and data imbalance problems. In this paper, a deep learning-based method combining Multi-gate Mixture-of-Experts (MMOE) and Synthetic Minority Over-sampling Technique (SMOTE) is proposed to address multimode imbalance fault diagnosis. Specifically, MMOE can identify modes and diagnose faults in each mode simultaneously. Furthermore, it investigates the common information of multiple modes by sharing network parameters. Moreover, SMOTE can address the fault imbalance problem by over-sampling minority fault samples. The experiment results show the performance improvements of multimode imbalance fault diagnosis using MMOE-SMOTE on the Tennessee Eastman (TE) Chemical benchmark process.

The maintenance and overhaul of power plant equipment largely depends on valuable knowledge accumulated during previous projects. This knowledge is often implicit and difficult to capture. This paper aims to address this challenge by proposing a deep learning based framework for the automatic extraction of knowledge from power plant maintenance reports in the form of knowledge graph (KG). Unlike other work, this framework can support effective text classification and Named Entity Recognition (NER) tasks for fire power plants reports. Specifically, we develop a TextCNN-GRU method with multi-attention to classify power plant maintenance text effectively, and use the BERT-BiLSTM-CRF model for NER tasks. To evaluate the proposed framework, we conduct experiments on a dataset collected from a realworld power plant, and the results obtained show outstanding performance. This work hence opens up opportunity for supporting intelligent maintenance of lire power plants using deep learning based methods and KG.

In order to improve the resource utilization rate of existing nuclear power data and promote workers to efficiently obtain the operation information of nuclear power units and assist them in fault diagnosis and maintenance decision-making, this paper constructs a knowledge graph question answering (KGQA) dataset in the field of nuclear power. The BEm-KGQA model based on the pre-trained language model and knowledge graph embedding method was proposed. Our model learns the embedded representation of the knowledge graph through BERT and fine-tunes the BERT model. In the question embedding stage, it learns the embedded representation of the question based on the fine-tuned BERT model. Through experiments, we demonstrate the effectiveness of the method over other models. In addition, this paper implements a nuclear power question answering system. Based on the question answering system, employees can learn about unit information and efficiently obtain information on unusual operating events of nuclear power.

Robotic Process Automation (RPA) has been widely used in business and enterprises to automate the processing of digital documents and collect information and acquire knowledge. Table structure reconstruction in scanned documents has been extensively studied as an essential application of RPA. However, the detection of table borders often ignores broken borders, which makes it unsuitable for natural scenes. To address this, our paper heavily employs a data augmentation approach to synthesize fake scanned documents to train our table-border semantic segmentation model. We propose a novel segmentation model for table borders based on semantic segmentation. We compare traditional morphology-based line detection algorithms with existing semantic segmentation-based approaches. The results indicate that our proposed algorithm can solve the frame line detection problem effectively, even for low-quality scanned images. Actual cases show that we can reconstruct the table’s structure and obtain the knowledge in the table.

Large language models (LLMs) exhibited powerful capability in various natural language processing tasks.This work focuses on exploring LLM performance on zero-shot information extraction, with a focus on the ChatGPT and named entity recognition (NER) task.Inspired by the remarkable reasoning capability of LLM on symbolic and arithmetic reasoning, we adapt the prevalent reasoning methods to NER and propose reasoning strategies tailored for NER.First, we explore a decomposed question-answering paradigm by breaking down the NER task into simpler subproblems by labels.Second, we propose syntactic augmentation to stimulate the model's intermediate thinking in two ways: syntactic prompting, which encourages the model to analyze the syntactic structure itself, and tool augmentation, which provides the model with the syntactic information generated by a parsing tool.Besides, we adapt self-consistency to NER by proposing a two-stage majority voting strategy, which first votes for the most consistent mentions, then the most consistent types.The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets, and on both domainspecific and general-domain scenarios.In addition, we present a comprehensive analysis of the error types with suggestions for optimization directions.We also verify the effectiveness of the proposed methods on the few-shot setting and other LLMs. 1 * Corresponding authors. 1 Code available at: https://github.com/Emma1066/ Zero-Shot-NER-with-ChatGPT Input Text: The player who temporarily ranks second is German athlete Bao Lizzo, with a total score of 355.02 points, slightly lower than Lanwei.Gold Label: {"German":"Geo-Political Entity", "Lanwei": "Person", "BaoꞏLizzo": "Person"} Vanilla Ans: {"German athlete Bao Lizzo": "Person", "Lanwei": "Person"} TS-SC Ans: {"BaoꞏLizzo": "Person": "Person", "Lanwei": "Person", "German": "Geo-Political Entity"} ----------------------------------

An image usually contains not only visual information but also higher-level semantic information. Nevertheless, previous computer vision algorithms, such as target detection and image classification, use only the visual features of the image alone. Recently, the explosion of scene graphs in computer vision has led to the challenge of generating structured scene graphs with rich semantic information. This paper proposes a one-stage query-based end-to-end Transformer model and generates scene graphs using the Hungarian matching algorithm. We develop an anti-bias reasoner module to reduce the impact of the unbalanced data distribution. Time-division training strategy is proposed to improve model training efficiency and speed up model convergence while improving model training performance. Experiments on the large-scale dataset Visual Genome were conducted in order to confirm the validity of our method. Compared with the existing state-of-the-art method, our method guarantees inference speed while maintaining acceptable performance and is more suitable for tasks with high real-time performance. Our work demonstrates that the one-stage method has great potential for exploration in scene graph generation.

As a biometric feature, handwritten signatures have various applications in finance, law, and business. The existing signature verification methods are mostly based on convolutional neural networks or Transformer based models. In this paper, we aim to implement writer independent offline handwritten signature verification by proposing an end-to-end method, Signature Verification Graph Convolutional Network (SigGCN). In SigGCN, signature images are first transformed into graphstructured data with additional position embedding to retain spatial information. The reason for this is that we expect graphstructured data to perform better in the capture of complicated relationships than CNN-based and Transformer based networks. We then use a multi-layer graph convolutional network to aggregate node information while updating the graph information. After obtaining the graph representation of signatures, efficient model training is performed using our proposed margin-based focal loss function to calculate the loss based on the Euclidean distance of two signatures. We conduct experiments on the CEDAR, BHSig260-Bengali, and BHSig260-Hindi datasets, and results obtained show that the proposed approach achieves remarkable performance and demonstrates great potential in solving real-world verification problems.

Fault diagnosis in an open world refers to the diagnosis tasks that need to cope with previously unknown faults in the online stage. It faces a great challenge yet to be addressed—that is, the online data of unknown faults may be classified as normal samples with a high probability. In this article, we develop an effective solution for this challenge by using supervised contrastive learning to learn a discriminative and compact embedding for the known normal situation and fault situations. Specifically, in addition to contrasting a given sample with other instances as is the case in conventional contrastive learning methods, our training scheme contrasts the normal samples with negative augmentations of themselves. The negative out-of-distribution data is generated by the Soft Brownian Offset sampling method to simulate the previously unknown faults. Computational experiments are conducted on the Tennessee Eastman Process benchmark dataset and a practical plasma etching process dataset. The proposed method achieves significant improvement compared with four existing methods under three open-set fault diagnosis circumstances, i.e., balanced open-set fault diagnosis, imbalanced fault diagnosis, and few-shot fault diagnosis. This demonstrates its great potentials in real world fault diagnosis applications.

Industry 4.0 has transformed the traditional production paradigm, leading to the applications of a range of industrial robotic arms for various purposes and in varied scenarios. The control and cooperation of these robots hold the key to swift reaction to uncertain needs from the market place, for which digital twins has become a promising technology. This paper aims to adding an intelligence dimension to digital twins by enabling learning capability for industrial robots using knowledge graph. Specifically, useful information about the operation scene and processing logic of a robot is collected and converted into graphical knowledge which will be accumulated and updated as a knowledge base for the operation of industrial robots. In our preliminary work, we evaluate the model on a simulation platform and demonstrate the potential of achieving learning capability for the robots through linking the physical world and the cyber world underpinned by knowledge graphs.

Researching is a process whereby a large amount of new and unstructured knowledge is created and accumulated. In this context, the capture of complex knowledge about detailed work and decision-making issues throughout a research project is very challenging for modern researchers. Knowledge graph technology can help machines better understand complicated relationships between entities, has great potential for helping researchers with organizing and automating such kinds of repetitive works, and even uncovering and providing new insights into related topics.This paper introduces a way to construct a research management platform by providing a profiling and query system visualized as a knowledge graph. With the scope of this platform being restricted to the research field, typical ontologies are proposed on different levels. For better and more meaningful visualization, specific modifications and improvements to the traditional knowledge graph structure are discussed. A prototype system that is under construction is then described based on the above work, with the extensive applications discussed.

Intelligent fault diagnosis is essential to guarantee the safe operation of industrial processes. And an important issue is how to develop a method to tackle the dilemma where we can only collect limited fault samples. In this paper, we propose a two-stage method based on supervised contrastive learning for imbalanced fault diagnosis tasks. We utilize the supervised contrastive learning technique as it has shown a powerful representation learning ability in previous works. The computational experiments on the Tennessee Eastman dataset show that our proposed two-stage method can achieve improved performance when compared to existing methods.

Computer supported cooperative work (CSCW) systems have been widely used to support teamwork in various fields such as design, education, research projects, etc. However, there is a great deal of knowledge generated directly or indirectly in the process of collaboration which is not well utilized and usually ignored instead of being reused and shared to help improve work efficiency. To bridge the gap between knowledge generation and utilization through CSCW, in this paper, we propose a knowledge mining based collaborative framework and first apply it to manufacturing value chains to achieve better work efficiency and reduce repetitive work in collaboration. Overall, this paper introduces the following novel insights and innovations: (1) we argue that there is a gap between the generated knowledge and the utilization of it during cooperative work; (2) a novel collaborative framework with knowledge mining approaches is proposed to bridge the gap; (3) a prototype system is further built and first applied to the manufacturing value chains. To the best of our knowledge, we are the first one to deal with knowledge reusing in CSCW of manufacturing value chains.

Physical knowledge is the foundation of most engineering fields in particular such as product design, analysis, and operation and maintenance. However, due to the complexity of physical concepts, laws, and calculations, students can be easily overwhelmed by the conceptual ideas in the process of learning physics. This paper proposes a new way for helping students grasp the logical relation between the physics knowledge points based on neural networks and knowledge graph technology. Specifically, we use Python scripts to collect the articles about physics knowledge on the Internet as the raw data. After removing the special characters and other irrelevant text, the rest of the data is passed to several neural networks based on BERT and ERNIE for their effective and efficient classification into seven kinds of physics knowledge. The experimental results show that using ERNIE-BERT for embedding and using RCNN for the downstream model achieve the best performance. Knowledge graph is used to build a tree structure of physics knowledge, holding the physics knowledge picked out by the neural networks under corresponding nodes.

Collaborative industrial robot is becoming more and more important in the manufacturing industry. Thanks for a variety of high-precision sensors integrated by the Cyber-Physical System, the collaborative industrial robot can model the working environment and know its own state in real time. Because of this, CPS enables the robot to plan a feasible path in a virtual simulation environment. In this paper, a two-dimensional working space and a three-dimensional working space is constructed and set as a virtual environment model constructed by CPS. Q-learning algorithm is used to plan a path in the working space. A feasible path is found by adjusting the number of iteration times of the algorithm. Further more, the learning rate α of the Q-learning algorithm is also adjusted and the results demonstrate that the increase of α will accelerate the convergence speed of the algorithm within the set range.

Open Knowledge Graph (OpenKG) link prediction is important for using OpenKGs in applications such as question answering and text comprehension. The noun phrases (NPs) and relation phrases in OpenKGs are not canonicalized, making OpenKG link prediction highly challenging. Existing methods addressing this problem infuse canonicalization information into knowledge graph embedding models. However, they still fail to fully exploit the semantics of NPs. First, two different NPs, even referring to the same entity, can carry different versions of information, which has been ignored by previous methods. Second, neighborhood information of NPs in OpenKGs has not been utilized, which contains abundant information for link prediction. Based on these observations, we propose the OpenKG Segmented Embedding (OKGSE) method. Specifically, to fully capture the dissimilarity of NPs belonging to the same cluster, we learn separate parts of embedding for both the NP cluster and NP. Meanwhile, we exploit neighborhood information by integrating graph context into the semantic matching score function. Extensive experiments across four benchmarks show that OKGSE can achieve state-of-the-art performance as well as effectively capture the unique semantics of each NP.

The maintenance of power plants highly relies upon precious knowledge and experience of handling faults, which is often stored in reports such as the event report. Simple string matching is the traditional means of retrieving relevant reports, and there is a failure of such methods in understanding the user's search intention properly. With a focus on improving the accuracy of information feedback, this work aims to develop a system of knowledge base question answering. Specifically, natural language processing is employed to improve question comprehension and information retrieval. The BiLSTM-CRF model and the fine-tuned BERT model are used to capture named entities and relations in the query. And the BM25 algorithm and the fine-tuned BERT model are combined to develop a scheme for better information retrieval. On this basis, the application interface of knowledge base question answering towards intelligent power plant maintenance is developed. With this question answering system, power plant operators can have better interaction with the knowledge base and improve collaboration.

In the process of transformer overhaul and maintenance in nuclear power plant, a large number of Chinese technical documents have been accumulated, including transformer structure and corresponding failure mode, failure cause analysis, maintenance test record, inspection and defect elimination record, fault problem and other information. It contains rich fault information, fault cause description and troubleshooting method and other key features of fault and maintenance, which is helpful for the maintenance of transformer To guide the operation and inspection work and fault diagnosis and analysis work. In CSCW and its related fields, artificial intelligence method and knowledge engineering technology, especially knowledge graphing technology and its intelligent application, have attracted extensive attention from academia and industry. Based on a series of data acquisition, analysis and algorithm flow, we have a deep understanding of the practical effect of applying intelligent algorithm directly to the construction of transformer fault konwledge graph and decision support application.

Imbalanced fault diagnosis becomes increasingly im-portant as the number of fault samples is relatively small in practical situations. Sparse auto-encoder(SAE) has been well addressed in fault diagnosis while it is not suitable for imbalanced fault diagnosis. Cost sensitive learning can be utilized to extend the sparse auto-encoder to cost sensitive sparse auto-encoder(CS-SAE). However, the class weights assigned to different classes are usually unknown in practice. So we propose to use particle swarm optimization(PSO) to optimize the class weights for cost sensitive sparse auto-encoder(PSO-CSSAE). The experiments have shown that the proposed approach consistently outperforms the state of the art on Tennesse Eastman(TE) dataset.

Smart Manufacturing is one of the key contents in the age of Industry 4.0, which helps to significantly improve the quality and efficiency of production. While the digital twin is the core technology in achieving smart manufacturing. Although the importance of Digital Twin is widely recognised, how to construct it and use it for smart manufacturing is not clear. To fill this gap, this paper proposes a systematic method in constructing a digital twin for the customized production, including 3D modelling, mechanism modelling, and real-time synchronization. A case study on constructing the digital twin for a customized furniture production factory has been used to demonstrate this method, with optimistic results proving its feasibility.

The assembly of customized and complex products entails the collaboration of heterogeneous devices and thus raises the need of effectively planning the assembly process according to the dynamic product and environment information. A cloud CPS-based intelligent assembly system is developed in this paper. Under the framework a product assembly model is introduced to describe the hierarchical relationships and mating features between subassemblies. The integrated assembly knowledge model including product structure, spatial position, mating features, and assembly process is presented. Sensory information collected from the real world and product model together form the assembly context. A two-step assembly knowledge reasoning process is then developed, where similar case matching finds the same or similar product structure from the existing assembly instance library, and priority rules guiding completes the final assembly sequence. A prototype system is developed and the gearbox assembly case validates the effectiveness of the proposed models.

Cloud computing technologies have enabled a new paradigm for intelligent manufacturing system which is powered by utilizing distributed resources, such as collaborative robots, simulation engines, advanced algorithms and human resources. As one of the key issues, the mechanism for online kinematics control of serial robotic manipulator presents speed challenge in the cloud-based system. In this research, a kinematics approximation model based on kernel ridge regression is developed for cloud manufacturing environment. To begin with, the model input is generated using trigonometric functions of rotation angles with permutation tricks which significantly reduces statistical error. Then, the approximation model is trained using kernel ridge regression with radial basis function, where both regularization and bandwidth of kernel have been optimized using grid-search. In addition, Universal Robot 10 is adapted as a collaborative robot example in simulation comparison experiments in order to evaluate the performance of the kinematics approximation model. As demonstrated in the experiment results, the proposed modelling approach can effectively support the cloud simulation paradigm and efficiently meet the real-time speed requirement in a distributed manufacturing environment.

The development of complex products entails the collaborative work of a multidisciplinary team and thus raises the need of effectively supporting knowledge creation and sharing in a collaborative and integrated working environment. A design knowledge model based on hypernetwork is proposed in this paper to facilitate knowledge management and collaborative reasoning in the design and development process. Specifically, the knowledge hypernetwork model is constructed with a designer network, a product network, an issue network and a knowledge unit network. The relationships between various nodes from different networks are identified and defined according to the node properties. On this basis, several topological characteristics of hypernetwork are analyzed and some statistical indicators of the design knowledge hypernetwork are defined. The Bayesian approach is applied to conduct the collaborative reasoning process whereby relevant knowledge units for different design entities are recommended according to the current design tasks and the issues to be resolved. A prototype system is developed and the BIW design case validate the effectiveness of the proposed hypernetwork-based model.

Cloud computing technologies have enabled a new paradigm for advanced product development powered by the provision and subscription of computational services in a multi-tenant distributed simulation environment. The description of computational resources and their optimal allocation among tenants with different requirements holds the key to implementing effective software systems for such a paradigm. To address this issue, a systematic framework for monitoring, analyzing and improving system performance is proposed in this research. Specifically, a radial basis function neural network is established to transform simulation tasks with abstract descriptions into specific resource requirements in terms of their quantities and qualities. Additionally, a novel mathematical model is constructed to represent the complex resource allocation process in a multi-tenant computing environment by considering priority-based tenant satisfaction, total computational cost and multi-level load balance. To achieve optimal resource allocation, an improved multi-objective genetic alqorithm is proposed based on the elitist archive and the K -means approaches. As demonstrated in a case study, the proposed framework and methods can effectively support the cloud simulation paradigm and efficiently meet tenants' computational requirements in a distributed environment.

This paper presents a novel evolutionary strategy for multiobjective optimization in which a population's evolution is guided by exploiting the geometric structure of its Pareto front. Specifically, the Pareto front of a particle population is regarded as a set of scattered points on which interpolation is performed using a geometric curve/surface model to construct a geometric parameter space. On this basis, the normal direction of this space can be obtained and the solutions located exactly in this direction are chosen as the guiding points. Then, the dominated solutions are processed by using a local optimization technique with the help of these guiding points. Particle populations can thus evolve toward optimal solutions with the guidance of such a geometric structure. The strategy is employed to develop a fast and robust algorithm based on correlation analysis for solving the optimization problems with more than three objectives. A number of computational experiments have been conducted to compare the algorithm to another three popular multiobjective algorithms. As demonstrated in the experiments, the proposed algorithm achieves remarkable performance in terms of the solutions obtained, robustness, and speed of convergence.

Collaborative design entails sharing knowledge in an effective way and from a variety of sources, which raises the need of developing technology-enabled systems that can improve decision making by streamlining complex analysis and synthesis processes and facilitating communication and negotiation in the design process. While previous research predominantly focuses on a particular aspect of design knowledge, an integrated method is required to develop such systems especially in the context that the amount of data has become huge. To address such a gap, this work aims to address the two views of design knowledge, namely codification and personalisation. The main idea is to facilitate the collaborative design process with a model-based solution which uses computational methods based on codified knowledge while incorporating rich personal experience of problem-solving for decision-making. This solution requires an effective knowledge representation that facilitates the integration of explicit and tacit knowledge in the computational processes. In addition, it relies on effective system integration during these processes. This paper reports the very first step towards such a solution, describing the system integration issues and the user interaction issues in the development of a prototype system.

Computer aided engineering analysis is widely applied in industry with the rapid development of computer power and simulation technology. In particular, the design and analysis of complex systems entails multidisciplinary collaborative modeling and thus requires enormous knowledge and experience. However, current work on managing design knowledge is mainly focused on capturing knowledge about the design solutions and processes while little work has been done on investigating the knowledge about the models and processes for engineering analysis. This research is aimed at addressing this gap by developing the methodology for effective modeling and re-using the knowledge generated in the engineering analysis process. Specifically, a system framework is proposed to identify the methodology for managing engineering analysis together with the key issues involved. A knowledge model is developed to describe complex engineering analysis problems as well as their solving processes. A case study has been undertaken and preliminary results show that the proposed solutions can effectively describe the models and processes for engineering analysis whilst this preliminary research opens up further opportunities for doing further work.

Multidisciplinary modeling and simulation of complex engineering systems are generally realized by an integrated approach in which the global system is decomposed into some coupling models of subsystems and then for collaborative computation. However, the coupling relationship of design variables between different disciplinary models usually increases the complexity of collaborative computation. It is a pivotal issue to solve these models in parallel for collaborative simulation. In this paper, numerical computation for multidisciplinary coupling models is analyzed. A method of variable-step collaborative simulation based on the local truncation error estimation is presented. A case study demonstrates the algorithm performance in both efficiency and accuracy.

Multidisciplinary models of complex products are developed by heterogeneous CAE tools in a distributed simulation environment. It is a key technology to integrate CAE models together, making them work collaboratively. In this paper, a modular method is proposed. First, a stage-by-stage modeling method is presented to reduce the interaction between modeling and simulation. Then, HLA and web service technology are employed to facilitate the simulation process. To fulfill the HLA based simulation, the development method of a general HLA adaptor is illustrated. A case study demonstrates the effectiveness of this approach.

Collaborative simulation is an important technology for complex engineering systems. The challenges of collaborative simulation are especially in model integration, interaction control and time advancement where a variety of collaborative disciplinary subsystems for time-synchronized running are handled dynamically in a distributed environment. In this paper, the formulized paradigm of multidisciplinary collaborative simulation for complex engineering systems is principally analyzed. A strategy for collaborative simulation in a distributed environment is proposed to implement an integrated framework leveraging HLA and web services technology. A minor-step-based interaction algorithm is presented to improve the precision. The interaction model definition and I/O data processing mechanism of the adaptor is studied. The prototype system and case studies are developed to demonstrate the effectiveness of this approach.

Simulation technology is widely applied in product development to make forecasts on the performance of design solutions for which it is difficult or not economical to develop physical prototypes. There are some circumstances, e.g. the development of mechatronic products, under which a number of simulations need to be used together. Therefore, it is necessary to develop a collaborative simulation platform which enables design engineers in a multidisciplinary team to develop, share, reuse, and integrate computational models. Difficulties in developing such a platform include using multiple simulation tools, the runtime integration of distributed models, the sharing of proprietary information, etc. Among these difficulties, run-time integration of distributed models is the one significantly affecting the accuracy, efficiency and stability of a simulation. This paper describes how multidisciplinary collaborative simulation problem can be solved, discusses different methods for run-time interaction, and presents a novel method that is proposed and implemented in a prototype system for collaborative simulation. Further evaluation on these methods shows that the novel method has better performance in terms of both accuracy and efficiency.

Collaborative simulation is proposed as a technique to support design evaluation with higher fidelity by addressing the interacting phenomena during the development of the complex product. To facilitate distributed simulation, high level architecture (HLA) was proposed and evolved as a standard with its specification, interfaces and data model. HLA provides simulation management for distributed component- based simulation and is identified as a candidate standard for collaborative simulation. Nevertheless, bottlenecks caused by the inherent deficiency of HLA restrict the applications of HLA-based collaborative simulation. In our presented research, we try to remedy the deficiency by integrating Web services technology to HLA-based simulation systems. Web services technology is adopted as the infrastructure for heterogeneous resources wrapping while HLA is the backbone for simulation scheduling. The framework and key issues for our solution are discussed in detail and a case study is performed. The prototype system and case study demonstrate that integrating web services with HLA is feasible in supporting collaborative simulation in the internet distributed environment with heterogeneous computing platforms.

Collaborative product development is concerned with various experiences, knowledge, teams, tools, processes, etc. Effective communication and cooperation of these factors are the key issues that challenge the current computational architectures. A service oriented approach is proposed by us to integrate available resources from a repository in open standard. In our solution, functionality of the any computational resource is encapsulated as web services with a variety of granularities. The services are defined, registered, consumed and orchestrated in service oriented architecture (SOA). Key issues of our solution are studied. A prototype system supporting collaborative simulation is implemented to verify the feasibility of the solution. The result shows that service oriented computing is promising for collaborative product development.

The development of complex product is essentially concerned with multidisciplinary knowledge and requires computer supported cooperation among the designers. A multidisciplinary collaborative modeling and simulation (M&S) technology is an effective approach for the design and development of the complex product. The challenge now faced is how to apply the multidisciplinary M&S in the systematic level of complex product development. The key issue is how to integrate the heterogeneous, autonomous software tools in a distributed environment. In this paper, we propose a high level architecture (HLA) and web services based framework for multidisciplinary collaborative design in a distributed environment. The process of multidisciplinary collaborative design project is principally analyzed. A modeling and simulation platform with the focus of multidisciplinary collaborative design is present. Besides its key technologies of HLA and web services based integration are studied. The interoperability issue to this research is also discussed.

This paper presents the design of a collaborative simulation framework based on HLA and Web service to make good use of the benefits of these two infrastructures. The framework leverages the Web-based technology to provide remote communication with user clients while using HLA for simulation traffic. Integration method, theoretical thoughts, technical considerations and up to date development of HLA and Web service are introduced in detail. To extend the framework towards supports for a knowledge-based complex product design system, an open and extensible environment based on the framework is put forward. Future work is discussed and the conclusion is given at the end of the paper