Robust object detection under varying weather conditions (e.g., rain, fog, and snow) presents significant challenges for industrial vision systems due to inherent visual degradations in manufacturing sites and outdoor facilities. While knowledge distillation offers promising potential by feature imitating and logit mimicking, existing methods face two critical limitations: first, inadequate mechanisms for effectively transferring localization capabilities in the presence of severe image degradation, and second, suboptimal strategies for identifying optimal distillation regions. To address these issues, we present a symmetric localization distillation loss based on the Jensen–Shannon divergence. Its mathematical characteristics, e.g., boundedness, symmetry, and gradient smoothness, enable robust preservation of spatial relationships and stabilize training processes. In addition, we present an adaptive label assignment strategy to select distillation regions, thus reducing the sparsity of positive samples during our knowledge distillation process. This work is the first one to apply knowledge distillation to object detection in inclement weather conditions. Extensive experiments on three challenging datasets show that our method improves the student model's object detection accuracy while maintaining its inference speed.

Topological semantic maps serve as an effective tool for the partially sighted or visually impaired (PSVI) in indoor navigation. Essential to their construction is the precise segmentation of floor plans. However, current techniques fall short in segmentation performance. To overcome this, we introduce a gated-dual-attention-based full-resolution network (GFNet) for floor plan segmentation. We leverage the inherent low-stage detailed features and intraclass-and-interclass contextual dependencies within floor plans to maximize segmentation effectiveness. Our proposed network incorporates modified residual blocks in early stages to maintain full-resolution capture with a low parameter count. We design and apply a novel gated dual attention (GDA) module that efficiently integrates channel and spatial contextual information to enhance local feature representation. This module improves the overall performance of our network while ensuring minimal parameter fluctuation. We also propose a 2-D deep supervision (TDDS) method to merge features from all stages, further enhancing the multilevel feature representation ability. Finally, a practical topological semantic mapping method for PSVI indoor navigation is introduced. All models are evaluated on the rasterized floor plan datasets R2V and R3D. Experimental results show that the proposed network achieves 2.21% and 2.74% mIoU improvements over the previous state-of-the-art method on R2V and R3D, respectively, for the challenging categories of walls and doors. This contributes to more accurate topological semantic mapping.

Achieving accurate segmentation in low-light scenes is challenging due to 1) severe domain shift encountered when models trained on daylight data are applied to such scenes and 2) lack of large-scale fine-grained labels in low-light conditions. A good idea is to use the generalization capabilities of segmentation foundation models like Segment Anything Model (SAM) to address the scarcity of annotated data. However, applying SAM to low-light scenes faces a severe domain shift issue due to the lack of inductive bias in effectively transforming low-light features into natural-light ones. To address this issue, we propose to adapt SAM for low-light scenes. To reduce the reliance on labels of low-light data, we develop a self-training method that makes SAM generate source-free predictions. To reduce the domain gap between low-light target data and SAM's natural-light trained data, we design a transformation head that enhances low-light features prior to the application of SAM. We further propose a domain shift compensation loss that trains our model to select a domain-adaptation-optimal illumination-enhanced feature map. Experimental results demonstrate that our method well outperforms the state of the art on the Dark Zurich and Nighttime Driving datasets. Code is available at https://github.com/HongminMu/SALS.

Large-scale outdoor navigation is essential for unmanned ground vehicles (UGVs), but despite significant advancements, they still face two key challenges in practical applications. The first one is how to ensure safe navigation in environments with dynamic and low-lying obstacles that LiDAR cannot detect. The second one is how to conduct the adaptive re-planning of target points while some of them are blocked by temporary obstacles. To address these challenges, this work proposes a Dynamic and Low-lying-obstacle Avoidance Navigation (DLAN) system to conduct perception, planning, and point correction for UGVs. To efficiently and accurately detect dynamic obstacles, it designs a lightweight Ensemble3D framework that integrates three fast but low-accuracy detection methods. A multi-criteria waypoint optimizer is used to assist UGVs in path planning. It ensures a balance between obstacle avoidance and path following. To adjust blocked target points through local re-planning, this work designs a checkpoint correction method. Extensive simulations and real-world experiments demonstrate that DLAN enables reliable navigation with high efficiency and robust obstacle avoidance in complex environments. More details can be found on our project homepage and video1.

Small object detection (SOD) given aerial images suffers from an information imbalance across different feature scales. This makes it extremely challenging to perform accurate SOD. Existing methods, e.g., Feature Pyramid Network (FPN)-based algorithms, focus on extracting high-resolution and low-resolution semantic features from different convolution layers. However, in deeper convolution layers, semantic feature misalignment and the loss of key information are inevitable. To tackle such issues, this work proposes a new encoder-decoder-based SOD framework with a Diffusion Model and Swin Transformer given aerial images. First, we reformulate an SOD task as a Noise-to-Box process. We then construct an encoder-decoder-based framework by using a diffusion model and Swin Transformer for dynamic bounding box generation. We introduce a decoupling training and inferencing strategy to recognize and locate small objects accurately. We finally evaluate the proposed framework on several public benchmarks. The experimental results well show its better SOD performance than the state of the art. Code is available at https://github.com/BrainPotter/EDSOD.

Adversarial patch defense has made significant progress recently, but defending against natural-looking patches remains a challenge due to their content-agnostic nature. We hypothesize that these patches exhibit position-related anomalies and are inspired by anomaly detection techniques. However, directly applying existing anomaly detection methods to patch detection behaves poorly because patches are a subset of anomalies, and using general anomalous features may introduce irrelevant anomalies. Additionally, anomaly detection datasets are primarily derived from industrial scenes, leading to out-of-distribution issues. To address these challenges, we propose a patch-agnostic defense method based on anomaly knowledge learning. It fine-tunes the Segment Anything Model in a self-supervised manner using an anomaly dataset, enabling the model’s image encoder to generate embeddings with enhanced activation for anomalous regions. It also designs a Cross Attention Patch Decoder based on cross-modal attention mechanisms to compute the mutual information between patch prediction probability maps and anomaly activation maps for patch localization. Our method shows strong performance on public datasets, and achieves a 16.9% mIoU improvement over the state-of-the-art in removing natural-looking patches with patch-to-target ratios over 0.6 on our constructed dataset.

This work tackles an integrated order batching, picker assignment, batch sequencing, and picker routing problem in warehouse environments. A Learning-Aided Iterated Local Search (LILS) is proposed to efficiently find its high-quality solutions. The main optimizer is iterated local search. A novel bidirectional long short-term memory network-embedded autoencoder, built through end-to-end unsupervised learning with an encoder and decoder, guides the search direction. To capture the implicit relationships among strongly-coupled subproblems, long short-term memory layers are incorporated in the encoder. A network-aided mutation operator is introduced to enhance global search in a low-dimensional feature space. In the decoder, low-order and high-fit subsolutions are identified and reconstructed to generate mutated offspring solutions using long short-term memory layers and a masking mechanism. To balance the exploration and exploitation of LILS, an information exchange method is developed. Numerical experiments show that LILS outperforms several existing methods in generating high-quality schedules within a reasonable time. Note to Practitioners—In a warehouse environment, an integrated optimization problem is often addressed by using heuristics due to limited computational resources. However, expedient heuristic rules tend to produce subpar results. While meta-heuristics can generate relatively better schedules, they are time-consuming, particularly for population-based algorithms that must iteratively evaluate fitness functions for numerous candidate solutions. In order to strike a balance between computational demands and solution-qualities, our approach combines machine learning techniques with meta-heuristics. Specifically, we integrate a bidirectional long short-term memory-based autoencoder into iterated local search to enhance the latter’s global optimization capability. The integration of machine learning and meta-heuristics enables efficient generation of superior schedules in a limited time. Various experimental results demonstrate that the proposed method significantly outperforms its recently-developed competitive peers, thus greatly facilitating the efficient operation of a smart warehouse.

This work addresses an energy-minimized deadline-constrained task scheduling problem in human-cyber-physical systems. It consists of three subproblems: processor allocation, task sequencing, and processor frequency scaling. A Learning-aided Evolutionary Algorithm (LEA) is proposed to efficiently find its reliable and high-quality solutions. It incorporates a bidirectional long short-term memory network-embedded autoencoder trained via end-to-end self-supervised learning. The model extracts the interconnections among the three strongly-coupled subproblems, enabling effective global search in a low-dimensional feature space. A parallel framework with two co-evolved subpopulations, one using the autoencoder and another undergoing regular evaluation in the original search space, is constructed. To balance LEA’s exploration and exploitation, a deep reinforcement learning-based search operator selection scheme is introduced, using a novel feedback-based reward function to guide operator selection for each subpopulation. Numerical experiments demonstrate that LEA surpasses several recently developed methods in finding high-quality schedules in a reasonable time.

This work considers a task scheduling problem with deadline constraints in human-cyber-physical systems. To find its energy-efficient schedules in a short time, an autoencoder-embedded iterated local search algorithm is proposed to solve it. Iterated local search is selected as a main scheduler. In order to handle real-time requirements and high computational load involved in the problem solution, a Long Short-Term Memory-based AutoEncoder model (LSTM-AE) is constructed to capture the relevant and complementary features of the considered problem. The model is used to find low-order and high-fit sub-solutions via unsupervised end-to-end learning, and generates promising solutions in an informative low-dimensional solution space. To further reduce computational burden, a two-stage optimization framework is constructed, which includes an off-line training phase and an online optimization one. The former trains LSTM-AE by using expert knowledge and historical data. The latter designs optimal resource allocation strategies to build a high-quality initial solution. Then, LSTM-AE-assisted local search operators are proposed and used to reform the initial solution and generate better ones. Various numerical experiments are performed to compare the proposed method with several classic heuristics and some recently-developed methods. The results show its superiority over them.Note to Practitioners—In human-cyber-physical systems, a task scheduling problem is usually solved by using heuristics due to the limited computational resources. Nevertheless, fast dispatching rules tend to perform poorly. Meta-heuristics can find a relatively high-quality schedule but are time-consuming, especially for a population-based algorithm that requires to evaluate a fitness function for many candidate solutions at each iteration. To balance computational burden and solution quality, our idea is to combine machine-learning methods with meta-heuristics. Specially, we integrate a long short-term memory-based autoencoder model into iterated local search to improve the latter’s optimization ability. The combination of meta-heuristics and machine learning techniques makes it possible to obtain a high-quality schedule for the concerned problems in a short time. Theoretic analysis and experimental results show that the proposed method well outperforms its competitive peers.

Real-time accurate visual object tracking (VOT) for quadrupedal robots is a great challenge when the scale or aspect ratio of moving objects vary. To overcome this challenge, existing methods apply anchor-based schemes that search a handcrafted space to locate moving objects. However, their performances are limited given complicated environments, especially when the speed of quadrupedal robots is relatively high. In this work, a newly designed VOT algorithm for a quadrupedal robot based on a Siamese network is introduced. First, a one-stage detector for locating moving objects is designed and applied. Then, position information of moving objects is fed into a newly designed Siamese adaptive network to estimate their scale and aspect ratio. For regressing bounding boxes of a target object, a box adaptive head with an asymmetric convolution (ACM) layer is newly proposed. The proposed approach is successfully used on a quadrupedal robot, which can accurately track a specific moving object in real-world complicated scenes.

In narrow environments with dynamic dense obstacles, it is difficult to find feasible paths for autonomous ground vehicles (AGVs) by using existing methods due to the strict security constraints on AGV movements. To overcome such difficulty, this work proposes an improved local path tracking algorithm based on model predictive control and a dynamic control barrier function with slack variable (MPC-DS). In this algorithm, slack variable is integrated with the control barrier function to convert the strict constraints to soft ones. To regulate the values of slack variable, a suitable penalty coefficient selected by using a series of comparative simulations is incorporated into MPC’s cost function. To test effectiveness of the proposed method, it is compared with three mainstream methods in environments with dense and sparse dynamic obstacles. Results of simulations and physical experiments show that AGV controlled by the proposed algorithm can avoid obstacles safely and efficiently in complex environments. It is worth noting that its use reduces energy consumption by 21.1% in comparison with the existing ones. To facilitate further research, our code is publicly available at https://github.com/BUCT-RobotLab417/MPC-DS

In the fields of robotic perception and computer vision, achieving accurate semantic segmentation of low-light or nighttime scenes is challenging. This is primarily due to the limited visibility of objects and the reduced texture and color contrasts among them. To address the issue of limited visibility, we propose a hierarchical gated convolution unit, which simultaneously expands the receptive field and restores edge texture. To address the issue of reduced texture among objects, we propose a dual closed-loop bipartite matching algorithm to establish a total loss function consisting of the unsupervised illumination enhancement loss and supervised intersection-over-union loss, thus enabling the joint minimization of both losses via the Hungarian algorithm. We thus achieve end-to-end training for a semantic segmentation network especially suitable for handling low-light scenes. Experimental results demonstrate that the proposed network surpasses existing methods on the Cityscapes dataset and notably outperforms state-of-the-art methods on both Dark Zurich and Nighttime Driving datasets.

Although remarkable advances have been achieved in generic object detection, small object detection (SOD) remains challenging owing to small objects’ information loss and noisy representation caused by their non-uniform distribution. Their limited width and height, scale variations, and redundant computation make SOD hard. To overcome them, this work proposes a new SOD method based on sparse convolutional network (SCNet) and Query Mechanism called QuerySOD. First, an extended feature pyramid network is constructed for extracting feature maps of small objects with more regional details. Then, a Sparse Head is neatly designed by using SCNet for accelerating the interfering speed and obtaining weights of each layer. After that, a Query Mechanism is innovatively introduced for harvesting the benefit of sparse value feature maps from the Sparse Head. QuerySOD is evaluated on public benchmarks including COCO and VisDrone. Finally, we apply it on ‘Jinghai’ unmanned survey vehicles and receive excellent SOD performance from this real-world application.

Topological semantic maps provide a practical solution to enhance indoor navigation for the Partially Sighted or Visually Impaired (PSVI). Segmenting indoor floor plans and extracting boundaries are key to constructing these maps. The existing methods exhibit low accuracy in segmentation. To achieve desired high segmentation accuracy, we introduce a Context-Enhanced Full-Resolution Network (CEFRN) for floor plan segmentation. It is designed to harness the shallow detailed features and inter-category contextual dependencies inherent in floor plans. CEFRN integrates modified residual blocks to capture the low-stage full-resolution features while maintaining its compactness. A position attention module is employed to refine the deep-stage contextual information. We also propose a two-dimensional deep supervision method to merge features from both stages, which significantly boosts the feature representation ability of CEFRN. Finally, a practical topological semantic mapping method for PSVI indoor navigation is introduced. Experimental results demonstrate that CEFRN’s segmentation accuracy well exceeds the state-of-the-art methods’. It can be used to well support accurate topological semantic mapping.

Image captioning is a multi-modal task that enables the transformation from scene images to natural language, providing valuable insights for visually impaired individuals to understand their environment. Therefore, its application to wearable navigation devices for visually impaired individuals holds immense potential. However, in practical applications, confusion between scene visuals and semantics, coupled with model complexity, often leads to performance degradation, resulting in inaccurate environmental interpretation. In light of this, we introduce a Lightweight De-confounding Transformer Network (LDTNet) for image captioning equipped with a Causal Adjustment module to eliminate confounders. Moreover, we design a Suppression Gate Unit that efficiently integrates fine-grained information from shallow features, while reducing the number of network layers to have a lightweight model. Experimental results demonstrate that our approach not only addresses the visual-semantic confusion issue effectively but also improves the response speed of wearable devices in comparison with the state of the art. Twenty volunteers are recruited to evaluate LDTNet’s efficacy in real-world settings in terms of both response speed and generated outputs by wearing the resulting assistive navigation devices. The outcomes well show its outstanding performance and great potential for visualy impaired individuals to use.

Cloud computing is a powerful paradigm that can provide high-quality computation services to customers. Because its energy consumption has a large effect on its service price, this study investigates how to minimize the energy consumption while achieving adequate response times for requested computations. Such a problem is formulated as a nonlinear integer program. By deriving a state transition equation, this problem is transformed into an unconventional 0–1 knapsack problem, and dynamic programming is then used to solve it. In addition to this solution, we develop an energy-efficient job accommodation scheme that can manage dynamic jobs with varying frequencies throughout a day. Unlike existing studies that abruptly switch off old virtual machines and create new ones for upcoming jobs, this scheme tries to accommodate them with current virtual machines, and new virtual machines are not created unless necessary. Conversely, when the workload declines, jobs on energy-inefficient servers are moved to other servers, such that some energy-inefficient servers can be switched off to save energy. This scheme adjusts the computing power adaptively and smoothly without lowering the system’s quality of service. Experimental results demonstrate that the proposed solution outperforms a particle swarm optimizer and two other heuristics in terms of accommodating jobs and saving energy consumption.

This work considers an extended flexible job-shop scheduling problem from a semiconductor manufacturing environment. To find its high-quality solution in a reasonable time, a learning-based genetic algorithm (LGA) that incorporates a parallel long short-term memory network-embedded autoencoder model is proposed. In it, genetic algorithm is selected as a main optimizer. A novel autoencoder model is trained offline via end-to-end unsupervised learning without relying on labeled data. This model captures the major linkages among decision variables and generates promising solutions in an informative low-dimensional space, striking a balance between computational efficiency and solution quality. To further improve its search ability, a co-evolving framework is designed, which includes both a network-embedded subpopulation and a regular one. The former focuses on its global search while the latter ensures LGA's convergence. An information exchange method between the two subpopulations balances global and local search, improving its overall optimization ability. This work conducts various numerical experiments to compare LGA with the CPLEX optimizer, several classical heuristics, and some popular methods. Results show that LGA outperforms its peers in finding high-quality solutions in a reasonable time.

To assist the Partially Sighted and Visually Impaired (PSVI), a variety of Obstacle Avoidance (OA) methods have been developed. These methods mostly use depth cameras for distance measurement in terms of perception, and communicate to users through voice broadcasts. However, due to insufficient detection accuracy and slow system response, they are difficult to apply to narrow and multi-pedestrian areas. To overcome this difficulty, this work aims to develop a dynamic OA system using an improved instance segmentation network for high-precision detection. To improve the segmentation accuracy of accessible paths for PSVI users, it proposes a new 2D convolution unit that couples multi-scale receptive fields of deep features. This unit focuses on the global context of an input image by constructing a hierarchical residual-like structure. To improve the efficiency of exploration, this work adopts a bidirectional A* algorithm with safety distance constraints to plan optimal paths for PSVI users, thus avoiding their trial-and-error path finding. To ensure safety, it proposes a collision avoidance algorithm based on regional safety analysis, which can generate and transmit timely vibration response to users. Experimental results demonstrate that our developed system can help PSVI users to pass through those challenging areas safely and effectively.

This paper considers a stochastic parallel machine scheduling problem in a just-in-time manufacturing context, in which its processing time can be described by a gamma or log-normal distribution. In order to obtain a high-performance schedule in a reasonable time, this work proposes a two-stage genetic algorithm with optimal computing budget allocation (OCBA) and improved Monte-Carlo Policy Evaluation (MCPE). In it, a genetic algorithm is selected as a main optimizer. An OCBA-based approach is developed to improve search efficiency, which is designed for two scenarios in a just-in-time manufacturing context. Different from most prior OCBA studies, this work considers that the stochastic processing time of jobs does not obey normal distribution. It extends the application area of OCBA by laying a theoretical foundation. A parameter control scheme based on MCPE is proposed, which aims to balance the global and local search in GA. To further enhance the efficiency and effectiveness of the proposed method, a two-stage framework is constructed. In the first stage, the performance is estimated roughly aiming at locating satisfactory solution regions. In the second stage, OCBA is incorporated to provide the reliable evaluation of excellent individuals. The theoretic interpretation of the proposed OCBA, and the convergence analysis results of the proposed method are presented. Various simulation results with benchmark and randomly generated cases validate that the proposed algorithm is more efficient and effective than several existing optimization algorithms. Note to Practitioners—A parallel machine scheduling problem under stochastic processing time is usually solved via meta-heuristic algorithms. However, their computational efficiency requires substantial improvement, especially for a stochastic optimization case that requires Monte Carlo sampling to estimate the actual objective function values in a precise manner. Most of them are parameter-sensitive, and choosing their proper parameters is highly challenging. For the first thorny issue, we develop an OCBA-based approach for determining the optimal numbers of simulations according to both prior knowledge and simulation results. In order to select proper control parameters of the proposed algorithm iteratively, we introduce a parameter control scheme based on MCPE. The combination of a meta-heuristic algorithm, OCBA and MCPE makes it possible to find high-quality solutions for the concerned scheduling problems in a short time. Theoretic analysis and numerical simulation results suggest that the proposed framework is valid and efficient. Hence, it can be readily applicable to practical systems, e.g., semiconductor manufacturing.

As vehicles integrate more and more autonomous driving functionalities, it becomes more and more important to use vehicular networks to fully guarantee the safety and real-time performance of on-board computing tasks. Current studies on vehicular networks pay much attention to the performance improvement of network communication and resource allocation, while ignoring the fact that automotive on-board computing tasks play a significant role in vehicle safety and need to be elegantly handled in vehicular networks. In this paper, we propose a hybrid scheduling framework for meeting all hard real-time task deadlines while minimizing soft real-time task deadline misses. In particular, the proposed scheduler is composed of some local schedulers and a global scheduler, where the former guarantees the schedulability of all hard real-time tasks, and the latter decides the assignment of soft real-time jobs dynamically online. Depending on the remaining processing capability of a vehicular network, arrival jobs are either assigned for further processing or discarded. An approach combining the utilization-based schedulability test and demand-supply analysis is proposed to effectively assign tasks to processors offline. To meet as many soft real-time task deadlines as possible, tasks' demand and supply bounds are computed at runtime, such that online execution information can be used to compensate for the scheduling loss with the worst-case assumption made by the offline test. Experimental results demonstrate that, compared to a genetic algorithm, our proposed approach needs far less computation to assign more tasks offline. The online scheduling also saves many soft real-time jobs that were to be dropped by the offline algorithm.

This work considers an extended version of flexible job-shop problem from a postprinting or semiconductor manufacturing environment, which needs a directed acyclic graph rather than a linear order to describe the precedences among operations. To obtain its reliable and high-quality schedule in a reasonable time, a learning-based cuckoo search (LCS) algorithm is presented. In it, cuckoo search is selected as an optimizer. To produce promising solutions in a high-dimensional solution space, a sparse autoencoder is introduced to compress a high-dimensional solution into an informative low-dimensional one. It extends the application area of autoencoder-embedded evolutionary optimization methods into combinational optimization by developing an improved one-hot encoding method. Then, in order to reveal the linkages among decision variables and enhance the explore ability of the proposed method, a factorization machine (FM) is used, for the first time, to capture the relevant and complementary features of population. Hence, a parallel framework involving three co-evolved subpopulations is constructed. The first one is an autoencoder embedded subpopulation, the second one is assisted by an FM, and the last one undergoes a regular iteration process. To balance the exploration and exploitation of the proposed framework and avoid unnecessary computation, a reinforcement learning algorithm is used to adaptively adjust the proportion of subpopulations and tune parameters of each subpopulation iteratively. Numerical simulations with benchmarks are performed to compare it with CPLEX, some classical heuristics, and several recently developed methods. The results shows that it well outperforms them.

Vehicular edge computing is a promising new computing paradigm that has lower service latency and higher bandwidth than cloud computing. However, the geographical dispersion of edge computing resources and the high dynamics of vehicles pose many challenges to its service provision. Aiming to minimize the energy consumption of vehicular edge computing servers, this article presents an adaptive scheduling approach for handling dynamic real-time computing requests. An auction-bid scheme is developed for deciding the roadside unit (RSU) to respond to the computing request, where the computing request is auctioned and the RSU with the least energy consumption gets the bid. This scheme works in a decentralized model that effectively reduces its implementation complexity. To process the computing request modeled as a directed acyclic graph (DAG) application, the upward rank value is used to decompose a DAG into individual tasks, and a deadline-aware queue jump algorithm is proposed to assign them to servers' queues in a specific RSU. A group scheduling scheme is developed to assign several applications as a group, for the purpose of searching for a better schedule. Extensive experiments are carried out to compare our proposed approach to some other heuristic and state-of-the-art approaches, and the results confirm the benefits of our proposed approach in terms of minimizing system energy consumption and providing a quick response to the computing request.

For tracking suspicious objects using intelligent robots, Multiple Object Tracking (MOT) has gained great attention. MOT is easily affected by long-term severe occlusion. This work proposes a joint MOT algorithm to handle such occlusion. Pairs of frames in complicated environments are taken as input. A center-based feature extraction framework is designed for precisely detecting objects and extracting their feature maps. A ConvGRU module is applied to learn permanent representations by using historical spatio-temporal information of objects. A Hungarian matching method is applied to match the detected objects and predicted predictions. The proposed algorithm is compared with several representative methods on two public multi-object tracking benchmarks. Furthermore, this work constructs a database with videos captured from street scenarios and uses it to test the proposed algorithm and its peers. Experimental results demonstrate that the proposed algorithm outperforms its peers, especially under long-term severe occlusion, thus advancing the field of MOT.

The 2-dimensional cost map has been widely used for the navigation of ground mobile robot. Although it is effective when the ground is flat, it becomes clumsy and ineffective when the ground has some slopes, where such slopes are often misjudged as the forbidden area by the cost map. For this reason, we propose a slope-adaptive navigation approach based on multilayer cost map in this paper. Instead of taking the point cloud of slope as obstacles, we actively construct a multi-layer cost map that takes slope information into the map in the stage of building environment map. A slope detection algorithm is developed to switch the cost map during the robot navigation. The slope is then considered as a passable road, only with extra cost. In the case that the slope leads the robot to a new floor, we adopt the Aruco code to switch the map information, such that the navigation can still keep working. Both simulation and real-world experimental results demonstrate the high effectiveness of our proposed approach.

This work considers a stochastic flexible job shop scheduling with limited extra resources and machine-dependent setup time in a semiconductor manufacturing environment, which is an NP-hard problem. In order to obtain its reliable and high-performance schedule in a reasonable time, a learning-based grey wolf optimizer is proposed. In it, an optimal computing budget allocation-based approach, which is designed for two scenarios from real manufacturing environments, is proposed to intelligently allocate computing budget and improve search efficiency. It extends the application area of optimal computing budget allocation by laying a theoretic foundation. Besides, to obtain proper control parameters iteratively, a reinforcement learning algorithm with a newly designed delay update strategy is used to build a parameter tuning scheme of a grey wolf optimizer. The scheme acts as a guide for balancing global and local search, thereby enhancing effectiveness of the proposed algorithm. The theoretic interpretation of the developed optimal computing budget allocation-based approach and the convergence analysis results of the proposed algorithm are presented. Various experiments with benchmarks and randomly generated cases are performed to compare it with several updated algorithms. The results shows its superiority over them. Note to Practitioners—Meta-heuristic are often deployed to solve semiconductor manufacturing scheduling problems. However, they face to two thorny issues when they face stochastic manufacturing environments. 1) their computational efficiency is quite low, thus requiring substantial improvement, since a stochastic optimization problem requires Monte Carlo sampling to estimate the actual objective function values in a precise manner; and 2) most of them are parameter-sensitive, and choosing their proper parameters is highly challenging in such environments. To address the first issue, we develop an optimal computing budget allocation-based method for deciding the optimal numbers of sampling times based on both prior knowledge and simulation results. To address the second one, we propose a reinforcement learning algorithm to self-adjust the parameters of our proposed method called Learning-based Grey Wolf Optimizer. In addition, we design a delay update strategy to enhance its robustness, and thus, a feasible and high-quality schedule can be founded in a short time for real-time scheduling problems. Theoretic proofs and experimental results show that the proposed method is effective and efficient. Consequently, it can be readily applicable to practical semiconductor manufacturing systems.

Cloud computing has become an important paradigm in which scalable resources such as CPU, memory, disk and IO devices can be provided to users to remotely process their applications. In a cloud computing platform, energy consumption accounts for a significant cost portion. This article thus aims to present an energy-efficient scheduling algorithm for processing a user application with a real-time requirement. This problem is formulated as a non-linear mixed integer programming problem. We start with providing an optimal closed-form solution to its relaxation problem that aims to minimize the energy consumption without considering real-time requirements. To meet real-time requirements, we propose how to adjust task placement and resource allocation by making a good tradeoff between energy consumption and task execution time. Lastly, we find two equivalent optimal resource allocation strategies once task placement has been done. We then propose to adjust the start time of task execution such that an application’s completion time can be further shortened. Experimental results on two real-case enchmarks and extensive synthetic applications demonstrate that our proposed method finds a schedule that generally has 30 and 20 percent less energy consumption than enhancement heterogeneous earliest finish time (E-HEFT) and genetic algorithm, respectively. Besides, the proposed method has a higher rate to successfully find a feasible schedule than them, and its computation time is close to E-HEFT’s, but far less than the genetic algorithm's.

This work investigates direction control and path following of a 3-D snake-like robot. In order to control such robots accurately, this work researches the relationships between its phase offsets of pitch joints and directions. A new direction control method is proposed for the robot based on these relationships. An adaptive path-following algorithm based on the line-of-sight guidance law is proposed and combined with the direction control method to steer the robot to move forward and along desired paths. Simulation and experimental results are presented to demonstrate the performances of the proposed 3-D model and control methods. They well outperform the classical and commonly used path-following method.

It is important to sufficiently guarantee an automotive system’s safety, because otherwise terrible consequences may happen. Generally the safety in an automotive system includes two aspects: reliability and timeliness. Previous studies have proposed many approaches to how to improve them. However, few of them consider the development cost along with their improvement. In this study, we aim to propose a method that can build a safety-guaranteed and development cost-minimized schedule for functionality modeled as a directed acyclic graph running on an automotive system. Unlike previous studies that tightly couple the development cost minimization with other requirements together, we start by building a schedule with the minimum development cost by ignoring safety requirement. Then, reliability and real-time requirements are subsequently taken into consideration. Together with automotive safety integrity level decomposition options provided by International Standard called ISO 26262, the decomposition is evaluated for each task to improve its safety, and tasks are then successively chosen to adjust the schedule, such that its safety can be maximized with incurring the least extra development cost. This procedure continues until a schedule that meets safety requirement is built. Experiments on a real-life automotive benchmark and extensive synthetic functionality demonstrate that our proposed heuristics outperform the state-of-the-art heuristic algorithm, and a typical intelligent optimization algorithm.

Today's large-scale parallel workflows are often processed on heterogeneous distributed computing platforms. From an economic perspective, computing resource providers should minimize the cost while offering high service quality. It has become well-recognized that energy consumption accounts for a large part of a computing system's total cost, and timeliness and reliability are two important service indicators. This work studies the problem of scheduling a parallel workflow that minimizes the system energy consumption under the constraints of response time and reliability. We first mathematically formulate this problem as a Non-linear Mixed Integer Programming problem. Since this problem is hard to solve directly, we present some highly-efficient heuristic solutions. Specifically, we first develop an algorithm that minimizes the schedule length while meeting reliability requirement, on top of which we propose a processor-merging algorithm and a slack time reclamation algorithm using a dynamic voltage frequency scaling (DVFS) technique to reduce energy consumption. The processor-merging algorithm tries to turn off some energy-inefficient processors such that energy consumption can be minimized. The DVFS technique is applied to scale down the processor frequency at both processor and task levels to reduce energy consumption. Experimental results on two real-life workflows and extensive synthetic parallel workflows demonstrate their effectiveness.

For the multi-robot path planning on a grid map, the widely adopted robot model assumes that its motion is deterministic once a path has been decided. However, this assumption is not quite realistic because some interferences such as noise, friction and inaccurate control input could disturb the robot motion, leading to a stochastic behavior. In this paper, we tackle the problem of planning a multi-robot path based on the robot probabilistic motion model. At the beginning, we model the robot action with several probability distributions, where the basic actions include going forward, turning left/right, going backward, and wait. We then extend A-star algorithm incorporating these actions such that an optimal path can be planned for a single robot. Based on this result, we apply conflict-based search to optimally plan path for a multi-robot system. Because probability calculation demands too much computation we simplify the conflict detection scheme and make it applicable for online practice.

Scheduling of complex manufacturing systems entails complicated constraints such as the mating operational one. Focusing on the real settings, this article considers an extended version of a flexible job shop problem that allows the precedence between the operations to be given by an arbitrary directed acyclic graph instead of a linear order. In order to obtain its reliable and high-performance schedule in a reasonable time, this article contributes a knowledge-based cuckoo search algorithm (KCSA) to the scheduling field. The proposed knowledge base is initially trained off-line on models before operations based on reinforcement learning and hybrid heuristics to store scheduling information and appropriate parameters. In its off-line training phase, the algorithm SARSA is used, for the first time, to build a self-adaptive parameter control scheme of the CS algorithm. In each iteration, the proposed knowledge base selects suitable parameters to ensure the desired diversification and intensification of population. It is then used to generate new solutions by probability sampling in a designed mutation phase. Moreover, it is updated via feedback information from a search process. Its influence on KCSA's performance is investigated and the time complexity of the KCSA is analyzed. The KCSA is validated with the benchmark and randomly generated cases. Various simulation experiments and comparisons between it and several popular methods are performed to validate its effectiveness.

This paper investigates a finite-frequency H-/H∞fault detection method for discrete-time T-S fuzzy systems with unmeasurable premise variables. To minimize the effect of uncertainties on system performance and maximize that of actuator faults on the generated residual, both the H∞disturbance attenuation index and finite-frequency H-fault sensitivity index are utilized. Since the premised variables are unmeasurable, the existing generalized Kalman-Yakubovich-Popov lemma cannot be directly extended to these nonlinear systems. In this paper, the conditions of allowing one to design the proposed H-/H∞fault detection observer are established and transformed into linear matrix inequalities. Some scalars and slack matrices are introduced to bring extra degrees of freedom in observer design. Finally, a single-link robotic manipulator model is utilized to illustrate that the proposed technique can detect faults with smaller amplitude than that required by a normal H∞observer technique.

Achieving the high accuracy of blind roads and crosswalks recognition is important for blind guiding equipment to help blind people sense the surrounding environment. A lightweight semantic segmentation network is proposed to quickly and accurately segment blind roads and crosswalks in a complex road environment. Specifically, a lightweight network with depthwise separable convolution as a component is used as a basic module to reduce the number of parameters of the model and increase the speed of semantic segmentation. In order to ensure the segmentation accuracy of the network, we use a densely connected atrous spatial pyramid pooling module to extract feature information of different angles and context feature modules to enhance the effectiveness of different levels of feature information fusion. To verify the effectiveness of the proposed method, we collect and produce a data set from a real environment, which contains two objects of blind roads and crosswalks1. Experimental results demonstrate that, compared to some state-of-the-art approaches, the proposed approach greatly improves the segmentation speed, while achieving better or similar accuracy, which shows that the proposed approach provides a better basis for the application of devices for guiding the blind.1https://github.com/qweawq/Blind-road-and-crosswalk-dataset

Snake-like robots move flexibly in complex environments due to their multiple degrees of freedom and various gaits. However, their existing 3-D models are not accurate enough, and most gaits are applicable to special environments only. This work investigates a 3-D model and designs hybrid 3-D gaits. In the proposed 3-D model, a robot is considered as a continuous beam system. Its normal reaction forces are computed based on the mechanics of materials. To improve the applicability of such robots to different terrains or tasks, this work designs hybrid 3-D gaits by mixing basic gaits in different parts of their bodies. Performances of hybrid gaits are analyzed based on extensive simulations. These gaits are compared with traditional gaits including lateral undulation, rectilinear, and sidewinding ones. Results of simulations and physical experiments are presented to demonstrate the performances of the proposed model and hybrid gaits of snake-like robots.

An important problem in multi-robot system is how to coordinate robot's motion such that each robot can complete its task without collision. Previous approaches assume that robots' motion is deterministic once their paths have been planned, which is however not realistic because random interferences such as noise, friction and inaccurate control input in real-life system could disturb the robot motion, leading to a stochastic behavior. In this paper, we take this stochastic behavior into account when planning the path for a heterogeneous multi-robot system. We assume that the motion time of a robot from a location to another can be modeled as a probability distribution. Every robot has its own probability distribution of motion time between any two neighbor locations. We develop a conflict-detection scheme for this model and propose using the conflict-based search algorithm via probability calculation to find the optimal path that minimizes the entire motion time. We also simplify this conflict detection such that our proposed approach is applicable online for a large-scale system. Experimental results demonstrate the high effectiveness of our proposed approaches.

Resource consumption cost minimization is important in embedded systems due to their limited resource and high computation load. Previous approaches apply either meta-heuristic algorithms such as genetic algorithm or simple heuristics to minimize the resource consumption cost, which, however, are computationally expensive and sometimes ineffective. In this article, we propose how to schedule a directed acyclic graph application with reliability requirement in a heterogeneous embedded system. The scheduling aim is to minimize system resource consumption cost. We start by finding a quasi-optimal solution that minimizes the resource consumption cost in ignorance of reliability. Then, we introduce an indicator that price the tradeoff between reliability and resource consumption cost, based on which we build a scheme that progressively tunes this schedule toward improving its reliability. We also explore an approach that updates this indicator in a lightweight way. Compared to several state-of-the-art approaches such as MRCRG, experimental results demonstrate that our approach constantly outperforms them, and its superiority becomes more significant with the enhancement of reliability requirement.

This paper presents a novel interval fault estimation approach by using zonotope technique for discrete-time linear parameter-varying systems in the presence of bounded parametric uncertainties, measured perturbation, and system disturbance. First, an augmented descriptor system is generated by using augmentation technique. Thus, the problem of interval fault estimation is transformed into the interval augmented state estimation. Then, an outer approximation of the new augmented state estimation domain is computed by using zonotope method. A zonotope should be consistent with the given outputs, perturbation, disturbance, and parametric uncertainties. Besides, it is minimized at each sampled time via an analytic formulation. Finally, a vehicle lateral dynamic nonlinear model is utilized to show the feasibility and effectiveness of the proposed zonotopic fault estimation technique.

Prioritized motion planning is an effective approach to decouple the motion coordination in multi-robot systems, in which priority assignment plays an important role on its performance. Previous approaches apply either a randomized search or a simple rule to prioritize robot motions, which are often computationally expensive and sometimes ineffective. In this paper, we present an effective approach to schedule robots movement with the aim of minimizing their task completion time. This approach first plans a time-optimal motion for each robot individually, then roughly prioritizes robots in the descending order of their motion-time, i.e., a robot is prioritized the highest if it has the largest motion-time, and vice versa. Then, we propose a heuristic to fine-tune robot priorities under the principle that the tuned priority shortens the task completion time. We also develop an asynchronous decentralized algorithm to accelerate the priority-tuning process. Compared to previous priority assignment approaches, our approach needs less computation time to generate a shorter task completion time in a multi-robot system.

Model-based reinforcement learning methods provide a promising direction for a range of automated applications, such as autonomous vehicles and legged robots, due to their sample-efficiency. However, their asymptotic performance is usually inferior compared to the state-of-the-art model-free reinforcement learning methods in locomotion control domains. One main challenge of model-based reinforcement learning is learning a dynamics model that is accurate enough for planning. This paper mitigates this issue by meta-reinforcement learning from an ensemble of dynamics models. A policy learns from dynamics models that hold different beliefs of a real environment. This procedure improves its adaptability and inaccuracy-tolerance ability. A proximal meta-reinforcement learning algorithm is introduced to improve computational efficiency and reduces variance of higher-order gradient estimation. A heteroscedastic noise is added to the training dataset, thus leading to a robust and efficient model learning. Subsequently, proximal meta-reinforcement learning maximizes the expected returns by sampling “imaginary” trajectories from the learned dynamics, which does not require real environment data and can be deployed on many servers in parallel to speed up the whole learning process. The aim of this work is to reduce the sample-complexity and computational cost of reinforcement learning in robot locomotion tasks. Simulation experiments show that the proposed algorithm achieves an asymptotic performance compared with the state-of-the-art model-free reinforcement learning methods with significantly fewer samples, which confirm our theoretical results.

A semiconductor final testing scheduling problem with multiresource constraints is considered in this paper, which is proved to be NP-hard. To minimize the makespan for this scheduling problem, a cuckoo search algorithm with reinforcement learning (RL) and surrogate modeling is presented. A parameter control scheme is proposed to ensure the desired diversification and intensification of population on the basis of RL, which uses the proportion of beneficial mutation as feedback information according to Rechenberg's 1/5 criterion. To reduce computational complexity, a surrogate model is employed to evaluate the relative ranking of solutions. A heuristic approach based on the relative ranking of encoding value and a modular function is proposed to convert continuous solutions obtained from Lévy flight into discrete ones. The computational complexity and convergence analysis results are presented. The proposed algorithm is validated with benchmark and randomly generated cases. Various simulation experiments and comparison between the proposed algorithm and several popular methods are performed to validate its effectiveness.

In this paper, the problem of path following for underactuated snake robots is investigated by using approximate dynamic programming and neural networks (NNs). The lateral undulatory gait of a snake robot is stabilized in a virtual holonomic constraint manifold through a partial feedback linearizing control law. Based on a dynamic compensator and Line-of-Sight guidance law, the path-following problem is transformed to a regulation problem of a nonlinear system with uncertainties. Subsequently, it is solved by an infinite horizon optimal control scheme using a single critic NN. A novel fluctuating learning algorithm is derived to approximate the associated cost function online and relax the initial stabilizing control requirement. The approximate optimal control input is derived by solving a modified Hamilton-Jacobi-Bellman equation. The conventional persistence of excitation condition is relaxed by using experience replay technique. The proposed control scheme ensures that all states of the snake robot are uniformly ultimate bounded which is analyzed by using the Lyapunov approach, and the tracking error asymptotically converges to a residual set. Simulation results are presented to verify the effectiveness of the proposed method.

In this paper, a mixed controller for solving the trajectory tracking and point stabilization problems of a mobile robot is presented, applying the integration of backstepping technique and neural dynamics. By introducing a virtual target point, the whole motion process is divided into two parts. The first one is employed to realize tracking control and the other one is adopted to implement point stabilization. Each part produces a feedback control law by using backstepping technique. Moreover, to solve the speed and torque jump problems and make the controller generate smooth and continuous signal when controllers switch, the neural dynamics model is integrated into the backstepping. The stability of the proposed control system is analyzed by using Lyapunov theory. Finally, simulation results are given to illustrate the effectiveness of the proposed control scheme.

In this paper, a sensor-based motion planning method for robot arm manipulators operating among unknown obstacles of arbitrary shape is presented. It can be applied to on-line collision avoidance with no prior knowledge of the obstacles. The sensitive skin is used to build a description of the robot’s surroundings. This approach is based on the configuration space but the construction of the C-obstacle surface is avoided. The motion planning algorithm consists of three phases. In each phase, the point automation moves along a specified plane, on which the mapping of the obstacles into configuration space can be simplified. Hence, the computation time is reduced and the algorithm can work in real time. The effectiveness of the proposed method is verified by a series of simulations.