The increasing automation of modern systems—across industry, healthcare, mobility, and beyond—has raised the demand for human reasoning and expertise, while alleviating the burden of repetitive tasks. This transformation is driving us toward Automation 5.0, a new paradigm aimed at unleashing human potential. Recently, the development of foundation models (FMs) has reinvigorated its realization, making it both urgent and critical to explore the concept of Automation 5.0 in this new era. In this article, we define Automation 5.0, discuss its significance, and emphasize its new world, thinking, and technology with the goal of achieving knowledge automation. A framework, based on business FMs, human-oriented operating systems, and scenarios engineering, is proposed, where biological, robotic, and digital humans work together in three modes: autonomous, parallel, and expert/emergency modes. Additionally, a diverse range of its scenarios and applications are summarized and discussed, such as Manufacturing 5.0, Healthcare 5.0, and Transportation 5.0. We believe that Automation 5.0 can drive the co-evolution of productivity and production relations across all domains, propelling society toward a “Safety, Security, Sustainability, Sensitivity, Service, Smartness (6S)” future.

Discrete data-based learning approaches have facilitated the wide applications of AI models, especially the notably favored foundation models. However, simply scaling the diversity and quantity of training data is still inadequate to achieve human-like thinking and action competency. A shift of learning paradigm from spatially–temporally discrete, weakly correlated, and noninteractive samples to spatially–temporally continuous, strongly correlated, and interactive scenarios is expected to go beyond the element-level understanding and promote the relation, trend, as well as situation awareness abilities of AI models. This article systematically structures the methodology of scenarios engineering (SE) and proposes a three-layer SE roadmap consisting of the scenarios development layer, scenarios organization layer, and scenarios cognition layer. This roadmap is designed to foster the flexible and efficient construction, organization, and utilization of scenarios. Building on this foundation and parallel intelligence, we introduce the framework of scenarios intelligence (SI) that leverages scenarios as next-generation data resources and microworld models to cultivate embodied AI agents, facilitating the development of descriptive, predictive, and prescriptive intelligence in tasks like perception, decision-making, and action. Experiments are conducted with unmanned aerial vehicles (UAVs) to illustrate the effectiveness of the proposed method in environmental understanding, risk assessment, and active perception.

Given the scarcity of real data and the time-intensive nature of labeling, current multi-agent perception models often rely on simulated sensor data for training and validation. However, perception performance deteriorates significantly due to domain gap between simulated and real data. Existing adaptation methods focus on domain-generalized feature extraction while neglecting multi-agent shift uncertainty and relational semantic loss. To address this issue, we propose a Selective Shift Domain Adaptation method in multi-agent collaborative perception, called SSDA. SSDA incorporates two essential components: the frequency-decoupled feature shift adjustment (FSA) and the entropy-driven staged adaptive alignment (SAA). To mitigate the relational semantic loss, the FSA is proposed to simplify the representation of correlation features and remove redundant information from the source domain, thereby mitigating interference for domain adversarial scenarios. To tackle the shift uncertainty, the SAA is designed to achieve adaptive alignment from global to local guided by information entropy, which dynamically adjusts weights for samples according to their level of uncertainty. The results demonstrate that the SSDA is significantly superior to the SOTA, achieving up to 7.35% improvements on [email protected].

Unmanned Aerial Vehicles (UAVs) are increasingly important in dynamic environments such as logistics transportation and disaster response. However, current tasks often rely on human operators to monitor aerial videos and make operational decisions. This mode of human-machine collaboration suffers from significant limitations in efficiency and adaptability. In this paper, we present AirVista-II—an end-to-end agentic system for embodied UAVs, designed to enable general-purpose semantic understanding and reasoning in dynamic scenes. The system integrates agent-based task identification and scheduling, multimodal perception mechanisms, and differentiated keyframe extraction strategies tailored for various temporal scenarios, enabling the efficient capture of critical scene information. Experimental results demonstrate that the proposed system achieves high-quality semantic understanding across diverse UAV-based dynamic scenarios under a zero-shot setting.

Isolated data islands are prevalent in intelligent automated optical inspection (AOI) systems, limiting the full utilization of data resources and impeding the potential of AOI systems. Establishing a collaborative ecology involving software providers, hardware manufacturers, and factories offers an encouraging solution to build a closed-loop data flow and achieve optimal data resource utilization. However, concerns about privacy issues, rights infringement, and threats from other participants present challenges in establishing an efficient and effective community. In this paper, we propose a novel framework, AOI-OPEN, which first creates a trustworthy AOI ecology to gather related entities with decentralized autonomous organization (DAO) mechanisms. Then, a parallel data pipeline is proposed to generate large-scale virtual samples from small-scale real data for AOI systems. Finally, federated learning (FL) is adopted to use the distributed data resources among multiple entities and build privacy-preserving big models. Experiments on defect classification tasks show that, with privacy preserved, AOI-OPEN greatly strengthens the utilization of distributed data resources and improves the accuracy of inspection models.

Recent years have witnessed remarkable advance-ments in entity alignment, which endeavors to identify entities that represent the same real-world objects across different knowledge graphs (KGs). Nonetheless, prevailing approaches predominantly operate within closed-domain scenarios, rendering them inadequate for handling unmatchable entities. To address this challenge, we propose a retrieval augmented large language model framework (RALLM) to leverage the reasoning capacities of large language models (LLMs) to achieve open-set entity alignment, which not only enables the identification of equivalent entities for matchable entities but also addresses the identification of unmatchable ones. Specifically, we propose a novel retrieval augmentation method that leverages both textual and structural information of entities to retrieve potential equivalent candidates. Subsequently, we employ an iterative process to prompt the LLM to discern the equivalence between the retrieved candidate entity and the entity requiring alignment. To mitigate issues related to many-to-one alignment prediction and enhance alignment efficacy, we devise a memory mechanism to store highly confident aligned entity pairs and provide reminders to the LLM when a candidate entity has been matched. Our experimental findings underscore the superior performance of RALLM, highlighting the potential of LLMs in facilitating open-set entity alignment tasks.

本文概述了基于基础模型和并行智能开发基础、基础设施机器人、机器人技术的初始步骤和基本框架,以及新型人工智能技术(如AlphaGO、ChatGPT和Sora)的潜在应用。

Replacing manual inspection, automated optical inspection (AOI) equipment is widely used in printed circuit board (PCB) factories for automatic PCB defect segmentation. However, parameter refinement of AOI devices has gradually become an efficiency bottleneck in AOI usage, posing a highly challenging task. Since a large number of AOI parameters and different types of inspected objects make timely proper parameter refinement for clear images quite difficult. Considering this, we propose the concept of parallel PCB inspection in cyber–physical–social systems (CPSSs). Based on artificial systems, computational experiments, and parallel execution (ACP) theory with automatic parameter identification and refinement, we perform descriptive intelligence to build an artificial imaging system, obtain knowledge about the mapping relationships of parameter settings and imaging results, and realize automatic parameter identification given image input; conduct predictive intelligence to obtain image quality assessment results and maximize quality score for refinement strategies; and carry out prescriptive intelligence to guide parameter refinement for better imaging. This system could guide engineers proactively with constructive suggestions on parameter refinement when imaging failures occur, greatly reducing the training cost of engineers while improving work efficiency and work quality. To validate that our parallel PCB inspection could perform automatic AOI results evaluation without human participation, we evaluate it on distortion-free and different distortion images and confirm image quality score is positively associated with segmentation accuracy.

Traditional convolutional neural networks (CNNs) share their kernels among all positions of the input, which may constrain the representation ability in feature extraction. Dynamic convolution proposes to generate different kernels for different inputs to improve the model capacity. However, the total parameters of the dynamic network can be significantly huge. In this article, we propose a lightweight dynamic convolution method to strengthen traditional CNNs with an affordable increase of total parameters and multiply-adds. Instead of generating the whole kernels directly or combining several static kernels, we choose to "look inside," learning the attention within convolutional kernels. An extra network is used to adjust the weights of kernels for every feature aggregation operation. By combining local and global contexts, the proposed approach can capture the variance among different samples, the variance in different positions of the feature maps, and the variance in different positions inside sliding windows. With a minor increase in the number of model parameters, remarkable improvements in image classification on CIFAR and ImageNet with multiple backbones have been obtained. Experiments on object detection also verify the effectiveness of the proposed method.

Federated learning is an emerging distributed machine learning paradigm that can break through data silos and make use of data from different clients in a secure way. However, for deep neural networks in federated learning, the models on clients may learn the same pattern with different weight distributions despite the same data distribution of local data sets, which limits the performance of neural networks after weight fusions. Therefore, in this article, we propose a homophily learning-based federated intelligence (HLFI) approach, where hierarchical federated learning strategy and dynamic elimination learning strategy are designed to alleviate the problem. The experiments on equipment failure prediction show that the proposed approach can improve the failure prediction F1-score up to 9.32%. Our approach also has good generalization capabilities and can be applied in other federated learning methods to improve the model performance.

To provide off-chain federations with complete privacy services to realize on-chain federated learning (FL), this article proposes a novel privacy framework for FL based on blockchain and smart contracts, named Artificial Identification. It consists of two modules: private peer-to-peer identification and private FL, using two scalable smart contracts to manage the identification and learning process, respectively. Based on Ethereum and interplenary file systems (IPFS), we implement our framework and comprehensively analyze its performance. Experiments show that the proposed framework has acceptable collaboration costs and offers advantages in terms of privacy, security, and decentralization. Furthermore, combined with radio frequency identification (RFID) technology, the framework has the potential to realize automatic on-chain identification and autonomous FL of machine clusters composed of Internet of Things (IoT) devices or distributed participants.

Federated learning has become an emerging hot research field in industry because of its ability to perform large-scale distributed learning while preserving data privacy. However, recent studies have shown that in the actual use of federated learning, there are device heterogeneity and data not identically and independently distributed (Non-IID) characteristics between client nodes, which will affect the effect of federated learning. In this work, we propose resilient reinforcement federated learning (R$^{2}$Fed), a R$^{2}$Fed method, which applies reinforcement learning to federated learning and uses reinforcement learning for weighted fusion of client models instead of average fusion. We conduct experiments on object detection, object classification, and sentiment classification tasks in the context of Non-IID and heterogeneity, and the experimental results show that the R$^{2}$Fed method outperforms traditional federated learning, increasing the average accuracy by 4.7%. Experiments also demonstrate that R$^{2}$Fed is resilient to federation attacks.

Rapid development of AI technologies has propelled the seamless integration of physical and cyber worlds with various kinds of online/offline information collected from millions of multimodal sensing systems. The complexity, diversity and uncertainty inherited in such systems, such as Intelligent Transportation Systems (ITSs), have gone far beyond human capacity of managing and controlling. Our team is among the first to propose the idea of utilizing the nearly unlimited computational resources in cyberspace to construct a bottom-up and top-down combined artificial ITSs for testing, experimenting, representation, verification, and validation of physical ITSs. Especially, the parallel transportation has been developed for safer, smarter, greener, and more reliable transportation services. After three decades of research and field studies, the DeCAST in Transverse, i.e., Decentralized/Distributed Autonomous Operations/Organizations (DAO) in transportation systems, has been envisioned. In this paper, we introduce its architecture, operational processes, software and hardware platforms, and real world applications. Specifically, a transportation foundation model driven by artificial transportation systems, parallel learning and federated intelligence, named TengYun, is outlined for DeCAST.

Foundation models are used to train a broad system of general data to build adaptations to new bottlenecks. Typically, they contain hundreds of billions of hyperparameters that have been trained with hundreds of gigabytes of data. However, this type of black-box vulnerability places foundation models at risk of data poisoning attacks that are designed to pass on misinformation or purposely introduce machine bias. Moreover, ordinary researchers have not been able to completely participate due to the rise in deployment standards. This study introduces the theoretical framework of scenarios engineering (SE) for building accessible and reliable foundation models in metaverse, namely, “SE-enabled foundation models in metaverse.” Particularly, the research framework comprises a six-layer architecture (infrastructure layer, operation layer, knowledge layer, intelligence layer, management layer, and interaction layer), which can provide controllability, trustworthiness, and interactivity for the foundation models in metaverse. This creates closed-loop, virtual–real, and human–machine environments that provides the best indices and goals for the foundation models, which allows us to fully validate and calibrate the corresponding models. Then, examples of use cases from the automotive industry are listed to provide transparency on the possible use and benefits of our approach. Finally, the open research topics of related frameworks are discussed.

Metaverses have caused significant changes in the industry and their academic foundation can be traced back to the term cyber–physical–social systems (CPSS), which was proposed in 2010. Radar is an important sensor in sensing systems that are widely applied in many fields, especially, in autonomous driving. To deal with the complex environment, smart radars with real-time information processing capabilities are required. Human factors play a critical role in the operation and management of radar systems, thus, digital twins’ radars in cyber–physical systems (CPS) are unable to achieve intelligence in CPSS due to an incomplete consideration of human involvement. For this consideration, we propose a novel framework of RadarVerses for smart radars in metaverses based on ACP-based parallel intelligence, which is also known as cyber–physical–social intelligence (CPSI). RadarVerses consist of five main parts which are physical radars, descriptive radars, predictive radars, prescriptive radars, and deep radars. To construct RadarVerses at the technical level, we introduce four main technical foundations: 1) communication technology; 2) scenarios engineering; 3) foundation models; and 4) digital workers. In addition, we also provide a case study about LiDARs’ predictive maintenance of accumulated snow in RadarVerses.

Industrial processes are typical cyber–physical–social systems (CPSSs), where the effective management of employees and the efficient control of machines play important roles. Traditional industries heavily rely on human labor and neglect the development of collection–utilization–transmission integrated information loops, thereby leading to high costs and low efficiency in operational procedures. To facilitate the natural interactions and smart operations for humans and machines, industrial foundation models (IFMs) based on metaverses are proposed in this article, serving as the operating systems of industrial parallel machines that provide sustainable data resources and scenarios for management and control experiments. On this basis, IFM comprised of vision foundation models, language foundation models, as well as operational foundation models, are constructed to manage resources in industrial parallel machines and provides comprehensive services for industrial procedures. On the one hand, IFM can efficiently manage various resources including computing power, digital assets, enterprise resources, and platform I/O via the proposed CPSS-based competing, sharing, scheduling, monitoring, allocating, and recovering mechanisms. On the other hand, imaginative intelligence, linguistic intelligence, and algorithmic intelligence can be achieved through vivid visualization of vision foundation models, natural conversations of language foundation models, and smart manipulation of operational foundation models. With the proposed IFM, cyber–physical–social intelligence (CPSI) can be achieved to enhance the efficient management and control of industrial processes.

Accurate and timely traffic flow prediction is crucial for intelligent transportation systems (ITS). Recent advances in graph-based neural networks have achieved promising prediction results. However, some challenges remain, especially regarding graph construction and the time complexity of models. In this paper, we propose a multi-stream feature fusion approach to extract and integrate rich features from traffic data and leverage a data-driven adjacent matrix instead of the distance-based matrix to construct graphs. We calculate the Spearman rank correlation coefficient between monitor stations to obtain the initial adjacent matrix and fine-tune it while training. As to the model, we construct a multi-stream feature fusion block (MFFB) module, which includes a three-channel network and the soft-attention mechanism. The three-channel networks are graph convolutional neural network (GCN), gated recurrent unit (GRU) and fully connected neural network (FNN), which are used to extract spatial, temporal and other features, respectively. The soft-attention mechanism is utilized to integrate the obtained features. The MFFB modules are stacked, and a fully connected layer and a convolutional layer are used to make predictions. We conduct experiments on two real-world traffic prediction tasks and verify that our proposed approach outperforms the state-of-the-art methods within an acceptable time complexity.

Varying density of point clouds increases the difficulty of 3D detection. In this paper, we present a context-aware dynamic network (CADNet) to capture the variance of density by considering both point context and semantic context. Point-level contexts are generated from original point clouds to enlarge the effective receptive filed. They are extracted around the voxelized pillars based on our extended voxelization method and processed with the context encoder in parallel with the pillar features. With a large perception range, we are able to capture the variance of features for potential objects and generate attentive spatial guidance to help adjust the strengths for different regions. In the region proposal network, considering the limited representation ability of traditional convolution where same kernels are shared among different samples and positions, we propose a decomposable dynamic convolutional layer to adapt to the variance of input features by learning from the local semantic context. It adaptively generates the position-dependent coefficients for multiple fixed kernels and combines them to convolve with local features. Based on our dynamic convolution, we design a dual-path convolution block to further improve the representation ability. We conduct experiments on KITTI dataset and the proposed CADNet has achieved superior performance of 3D detection outperforming SECOND and PointPillars by a large margin at the speed of 30 FPS.

Welcome to the second issue of IEEE Transactions on Computational Social Systems (TCSS) this year. First, I am grateful to report that, as of February 7, 2021, theCitescoreof TCSS has leapfrogged back to 5.8, a new high, which indicates the high quality and relevance of IEEE TCSS in the field of social computing and computational social systems research. Many thanks to all of you for your great effort and support.

Federated ecology can provide an effective solution forthe serious isolated data island issues caused by data privacy protection and information security requirements in the era of artificial intelligence (AI). As the data foundation of federated ecology, federated data include the data of all the nodes in the federation, as wellas their storage, computation, and communication resources. For privacy-preserving, federated data are divided into private data and non-private data, and through the federated control of these data, data federalization can be realized. In data-driven AI technologies, federated data play an important role, and it can help realize effective data retrieval, pre-processing, processing, mining, and visualization for AI-based applications. It can also provide effective solutionsfor the dilemmas faced by AI technologies, such as training AI models without sufficient data, increasing the generality of AI models for different application scenarios and establishing a unified processing workflow for data security and privacy control in AI-based applications.

Object instance segmentation in traffic scenes is an important research topic. For training instance segmentation models, synthetic data can potentially complement real data, alleviating manual effort on annotating real images. However, the data distribution discrepancy between synthetic data and real data hampers the wide applications of synthetic data. In light of that, we propose a virtual-real interaction method for object instance segmentation. This method works over synthetic images with accurate annotations and real images without any labels. The virtual-real interaction guides the model to learn useful information from synthetic data while keeping consistent with real data. We first analyze the data distribution discrepancy from a probabilistic perspective, and divide it into image-level and instance-level discrepancies. Then, we design two components to align these discrepancies, i.e., global-level alignment and local-level alignment. Furthermore, a consistency alignment component is proposed to encourage the consistency between the global-level and the local-level alignment components. We evaluate the proposed approach on the real Cityscapes dataset by adapting from virtual SYNTHIA, Virtual KITTI, and VIPER datasets. The experimental results demonstrate that it achieves significantly better performance than state-of-the-art methods.

We propose Mask SSD, an efficient and effective approach to address the challenging instance segmentation task. Based on a single-shot detector, Mask SSD detects all instances in an image and marks the pixels that belong to each instance. It consists of a detection subnetwork that predicts object categories and bounding box locations, and an instance-level segmentation subnetwork that generates the foreground mask for each instance. In the detection subnetwork, multi-scale and feedback features from different layers are used to better represent objects of various sizes and provide high-level semantic information. Then, we adopt an assistant classification network to guide per-class score prediction, which consists of objectness prior and category likelihood. The instance-level segmentation subnetwork outputs pixel-wise segmentation for each detection while providing the multi-scale and feedback features from different layers as input. These two subnetworks are jointly optimized by a multi-task loss function, which renders Mask SSD direct prediction on detection and segmentation results. We conduct extensive experiments on PASCAL VOC, SBD, and MS COCO datasets to evaluate the performance of Mask SSD. Experimental results verify that as compared with state-of-the-art approaches, our proposed method has a comparable precision with less speed overhead.

Object instance segmentation can achieve preferable results, powered with sufficient labeled training data. However, it is time-consuming for manually labeling, leading to the lack of large-scale diversified datasets with accurate instance segmentation annotations. Exploiting the synthetic data is a very promising solution except for domain distribution mismatch between synthetic dataset and real dataset. In this paper, we propose a synthetic-to-real domain adaptation method for object instance segmentation. At first, this approach is trained to generate object detection and segmentation using annotated data from synthetic dataset. Then, a feature adaptation module (FAM) is applied to reduce data distribution mismatch between synthetic dataset and real dataset. The FAM performs domain adaptation from three different aspects: global-level base feature adaptation module, local-level instance feature adaptation module, and subtle-level mask feature adaptation module. It is implemented based on novel discriminator networks with adversarial learning. The three modules of FAM have positive effects on improving the performance when adapting from synthetic to real scenes. We evaluate the proposed approach on Cityscapes dataset by adapting from Virtual KITTI and SYNTHIA datasets. The results show that it achieves a significantly better performance over the state-of-the-art methods.

Dataset plays an essential role in the training and testing of traffic vision algorithms. However, the collection and annotation of images from the real world is time-consuming, labor-intensive, and error-prone. Therefore, more and more researchers have begun to explore the virtual dataset, to overcome the disadvantages of real datasets. In this paper, we propose a systematic method to construct large-scale artificial scenes and collect a new virtual dataset (named “ParallelEye”) for the traffic vision research. The Unity3D rendering software is used to simulate environmental changes in the artificial scenes and generate ground-truth labels automatically, including semantic/instance segmentation, object bounding boxes, and so on. In addition, we utilize ParallelEye in combination with real datasets to conduct experiments. The experimental results show the inclusion of virtual data helps to enhance the per-class accuracy in object detection and semantic segmentation. Meanwhile, it is also illustrated that the virtual data with controllable imaging conditions can be used to design evaluation experiments flexibly.