Multi-modal image fusion aims to integrate complementary cues from different modalities into a single image, facilitating downstream tasks such as object detection. However, input image pairs are often misaligned due to rigid or non-rigid deformations in practical scenarios. Such misregistration introduces structural distortions and visual artifacts, reducing the reliability of the fused results and limiting their effectiveness for downstream vision applications. While existing methods demonstrate satisfactory results under specific deformation scenarios, they exhibit limited generalization to diverse and severe misregistrations. To this end, this study proposes a unified multiple-in-one flow Mamba framework for registering various image deformations and generating high-quality fused results. Specifically, a hierarchical flow Mamba is designed to model rigid and non-rigid flow fields and enhance adaptability to complex deformations by progressively refining misaligned features. To better distinguish between rigid and non-rigid types, a flow field classifier predicts rigid/non-rigid categories and provides prompts for high-level feature modulation. Furthermore, a flow-guided fusion Mamba module is developed to aggregate aligned multi-level modality features and generate fused images, while an iterative training strategy enables collaborative optimization by using fusion outputs to refine flow estimation. Experiments across three representative modality tasks demonstrate that the proposed method delivers superior fusion performance while maintaining applicability to object detection. The code will be available at: https://github.com/BOYang-pro/MOFM.

In complex industrial production processes, accurate multivariate time series forecasting is essential for operational control, process optimization, and safety assurance. However, the inherent nonstationarity and multiscale characteristics of industrial time series present a significant challenge to achieving accurate predictions. To this end, this article proposes a novel method for multivariate nonstationary industrial time series forecasting, which integrates frequency-domain decomposition with season-trend bidirectional interaction. First, a learnable frequency-domain decomposition module is designed to decouple the input sequences adaptively into distinct components that possess different frequency properties. Second, a bidirectional gated interaction module is developed, which facilitates dynamic and controlled information exchange between the seasonal and trend encoding branches. Furthermore, a loss function incorporating trend consistency constraints is established to enhance the model's capability in capturing the dynamic morphologies of the sequences. Experimental results on multiple real-world industrial datasets demonstrate that our model outperforms existing state-of-the-art methods on three industrial datasets, thereby validating its effectiveness and practical value in forecasting complex industrial time series.

Confidence calibration of classification models is a technique to estimate the true posterior probability of the predicted class, which is critical for ensuring reliable decision-making in practical applications. Existing confidence calibration methods mostly use statistical techniques to estimate the calibration curve from data or fit a user-defined calibration function, but often overlook fully mining and utilizing the prior distribution behind the calibration curve. However, a well-informed prior distribution can provide valuable insights beyond the empirical data under the limited data or low-density regions of confidence scores. To fill this gap, this paper proposes a new method that integrates the prior distribution behind the calibration curve with empirical data to estimate a continuous calibration curve, which is realized by modeling the sampling process of calibration data as a binomial process and maximizing the likelihood function of the binomial process. We prove that the calibration curve estimating method is Lipschitz continuous with respect to data distribution and requires smaller sample sizes than histogram binning. Also, a new calibration metric has been designed, leveraging the estimated calibration curve to estimate the true calibration error, and it has been proven to be a consistent calibration measure. Furthermore, realistic calibration datasets can be generated by the binomial process modeling from a preset true calibration curve and confidence score distribution, which can serve as a benchmark to measure and compare the discrepancy between existing calibration metrics and the true calibration error. The effectiveness of our calibration method and metric are verified in real-world and simulated data. We believe our exploration of integrating prior distributions with empirical data will guide the development of better-calibrated models, contributing to trustworthy AI.

The occlusion problem has consistently posed a significant challenge in the field of segmentation. Most existing segmentation methods require additional annotations and fail to capture the contour information of occluded regions, thus not truly addressing the occlusion issue. Although segmentation tasks involving particle objects also suffer from occlusion problems, the homogeneity of particle objects offers new possibilities for overcoming this challenge. In this paper, we propose an occlusion segmentation framework for particle objects that does not require additional annotations. This framework only necessitates instance-level segmentation labels to obtain complete contour information of particle objects, including occluded regions. First, we decompose the occlusion segmentation task into a generic instance segmentation task and an occlusion repair task for occluded objects. Then, in order to train the occlusion repair model with only instance segmentation-level labels, we quantitatively analyze the occlusion phenomenon, including the mathematical descriptions of occlusion relationships, degrees, and distributions. Next, we geometrically transform and layer overlay the unobscured samples to construct occlusion samples containing labeling information of the occluded regions. These sample sets are used to train a generative model that predicts the contour information of occluded regions. Finally, we fine-tune or post-process the pre-segmentation model with the particle objects containing restored complete contour information to achieve the final occlusion segmentation. We conducted extensive ablation experiments on both the ore-particle dataset and publicly available cell-particle datasets. The experimental results validate the effectiveness, accuracy, and generalizability of our method. Note to Practitioners—Particle segmentation has been faced with the occlusion problem. In this paper, inspired by the similarity between particle objects, we propose a self-supervised occlusion segmentation framework that does not require additional annotation of occlusion layers. Our approach requires only instance segmentation level annotation without more complex additional manual annotation, which is crucial for practical applications. In addition, we decouple the complex occlusion relation modeling into a binary classification problem without knowing precisely the occlusion hierarchy between particles, which further reduces the difficulty of practical applications. Then, we also propose shading transformations to characterize the inter-particle shading distribution to construct shading sample sets from existing samples. Finally, we use these learned and constructed occlusion sample sets to pre-train the generative model for regenerating the occluded objects to complete the final occlusion segmentation. Although in a generic segmentation task, our approach may have some limitations because the segmented objects may not have an apparent similarity. However, our approach using self-supervision and the objects’ properties provides valuable ideas for solving the occlusion problem. In the future, we will solve the occlusion problem regarding the properties of each class of objects rather than just considering the similarity among particle objects.

Ensuring the accurate recognition of blast furnace (BF) conditions is crucial for maintaining the stability of blast furnace ironmaking process (BFIP). However, many research frequently overlooks the comprehensive utilization of the closely correlated multisource heterogeneous data (MHD) of BFIP, such as time series and images, resulting in an incomplete understanding of BF conditions and an unsatisfactory accuracy of condition recognition. Therefore, this article proposes a novel BF condition recognition method by incorporating both time series and images of BFIP. First, a data alignment model using the Gaussian functions is proposed to align time series with images. Then, an innovative Time Series—Image jointly driven deep neural Network (TSIN) is established to extract and fuse features from MHDs. Subsequently, a dual-layer residual-connected module is designed to capture the correlations between MHDs. Furthermore, a stability evaluation metric is devised to quantify the stability of TSIN during the model training process. Industrial experiments demonstrate that the proposed method, with an average recognition accuracy of 96.99%, could effectively recognize abnormal conditions, such as channeling, hanging, slipping, and collapsing, providing practical guidance for on-site workers to monitor and regulate the BFIP.

Confidence calibration in classification models is a vital technique for accurately estimating the posterior probabilities of predicted results, which is crucial for assessing the likelihood of correct decisions in real-world applications. Class imbalance data, which biases the model's learning and subsequently skews predicted posterior probabilities, makes confidence calibration more challenging. Especially for underrepresented classes, which are often more important and tend to have higher uncertainty, confidence calibration is more complex and essential. Unlike previous surveys that typically separately investigate confidence calibration or class imbalance, this article comprehensively investigates confidence calibration methods for deep learning-based classification models under class imbalance. First, the problem of confidence calibration under class imbalance data is outlined. Second, this article explores the impact of class imbalance data on confidence calibration in theory, providing some explanations for empirical findings in existing studies. Third, this article reviews 60 state-of-the-art confidence calibration methods under class imbalance data, divides these methods into six groups according to method differences, and systematically compares seven properties to evaluate their superiority. Then, some commonly used and emerging evaluation methodology are summarized, including public datasets and evaluation metrics. Subsequently, this article performs necessary comparative experiments to provide better guidelines and insights to the readership. Finally, we discuss several application fields and promising research directions that serve as a guideline for future studies.

Edge preserving filter is the basis of many computational photography and image processing. This can be achieved by global optimization method or local filtering method. Generally, the filtering results of global optimization methods are better than that of local filtering methods, and local filtering methods usually run much faster than global optimization methods. In this paper, a globally optimized method called iterative self-guided image filter (isGIF) is extended based on the assumptions of the guided image filter (GIF), which can produce high-quality edge-preserving filtering results by using the input image itself as the guidance image. Some comparisons with other edge-aware filters are presented to show the advantages of our method. Extensive experiments demonstrate that our filter generates images with better visual quality, while reducing/avoiding halo artifacts in the final image, and the running time is competitive.

Online assessment of the molten iron quality of a blast furnace ironmaking process strongly depends on reliable measurement of silicon (Si) content. In this article, we propose a novel cooperative strategy to train the data-driven prediction model and estimate variable time delay (VTD) values. First, a dynamic deep network based on stacked denoising autoencoders (D-SDAE) with the ability to describe process nonlinearity and time-varying is designed for Si estimation. Then, time delay between the process variables and Si, which may disrupt the data distribution pattern (i.e., the real input–output relationship), is recovered in the modeling process. It is an overlooked data feature that occurs due to material transportation times, installation location, and the analysis period of sampling equipment. The VTD values are considered to be the parameters of the D-SDAE model and obtained in the training process by the proposed double-scale collaborative search particle swarm optimization algorithm. The effectiveness of the proposed VTD-based D-SDAE model is validated in a numerical simulation and an industrial ironmaking plant, and a marked improvement in the prediction performance is achieved when VTD information is considered.

Charging operation is an essential regulatory measure at the top of blast furnaces. Accurately predicting the burden surface shape (BSS) and reasonably adjusting the charging matrix (CM) are vital for achieving precise charging operation. However, the existing methods neglect the influence of the burden motion state, which greatly limits prediction accuracy of BSS, and complex constraints of CM further cause the lack of effective CM optimization strategies. To address these challenges, a BSS prediction model and a CM optimization strategy are innovatively proposed in this article. First, the burden motion state under different accumulation behaviors is introduced into the BSS modeling, and the iterative solution of BSS is realized with the volume conservation constraint. Then, a hybrid constraint-handling mechanism consisting of the infeasible solution rejection method and the penalty function method is proposed, which is used to develop a CM optimization model. Finally, an improved algorithm, named dynamic heterogeneous multiswarm particle swarm optimization, is designed to solve the optimal CM (i.e., the best particle in the population). The prediction results of BSS model are highly similar to those of the model based on discrete element method, and the effectiveness of CM optimization strategy is validated in two simulation cases.

Making proper decision online in complex environment during the blast furnace (BF) operation is a key factor in achieving long-term success and profitability in the steel manufacturing industry. Regulatory lags, ore source uncertainty, and continuous decision requirement make it a challenging task. Recently, reinforcement learning (RL) has demonstrated state-of-the-art performance in various sequential decision-making problems. However, the strict safety requirements make it impossible to explore optimal decisions through online trial and error. Therefore, this article proposes a novel offline RL approach designed to ensure safety, maximize return, and address issues of partially observed states. Specifically, it utilizes an off-policy actor-critic framework to infer the optimal decision from expert operation trajectories. The "actor" in this framework is jointly trained by the supervision and evaluation signals to make decision with low risk and high return. Furthermore, we investigate a recurrent version of the actor and critic networks to better capture the complete observations, which solves the partially observed Markov decision process (POMDP) arising from sensor limitations. Verification within the BF smelting process demonstrates the improvements of the proposed algorithm in performance, i.e., safety and return.

Accurately measuring the slag-iron ratio (SIR) of the molten iron flow is of great significance for efficient slag-iron emission and safe production in the ironmaking process. Slag-iron recognition is challenging due to the high temperature, intense radiation, and heavy dust in the casting field. Efficient and intelligent online slag-iron recognition is urgently needed in ironmaking enterprises. To this end, this article innovatively presents a vision-driven slag-iron recognition method based on multiscale neighborhood adaptive clustering (MNAC) to achieve continuous high-precision SIR measurement of molten iron flow. First, a shortwave infrared high-speed imaging (SWIHI) system is designed and deployed to capture high-definition molten iron flow images. Then, a multialgorithm fusion key frame and region of interest (ROI) acquisition strategy is proposed to achieve stable acquisition of molten iron flow boundaries and prevent the ROI from being occluded by dust and objects. Finally, a novel multiscale neighborhood slag-iron recognition strategy and an adaptive density peak clustering algorithm are presented to achieve high-precision slag-iron recognition through the kernel density estimation of the membership of a single pixel in multiple neighborhoods. Industrial experiments and applications indicate that the proposed method can achieve slag-iron recognition continuously and accurately to provide reliable SIR data for blast furnace experts.

Process modeling and intelligent decision-making play crucial roles in optimizing and controlling sintering processes. To tackle these tasks, we propose a novel approach that leverages a temporal convolutional network guided by working condition knowledge (WKGA-TCN) to predict the sintering condition. By incorporating sequential features, the model effectively captures the varying importance of different features. Subsequently, we simulate the state of the sintering process using our proposed method. To further enhance operational optimization, we introduce an intelligent decision-making strategy that combines dynamic matching of operating modes and a genetic algorithm. This strategy leverages historical data to learn from excellent operation cases while considering the inherent uncertainty in process data. Simultaneously, a model update strategy employing an event-triggered mechanism is introduced to guarantee the process model's comprehensive simulation of the current sintering process dynamics. Finally, we assess the effectiveness of the proposed method by applying it to actual industrial process data.

Unlike general image segmentation, highly complex particle images have significant challenges in labeling and segmentation due to the information occlusion and texture disturbance. Aiming at the highly under-segmentation problem caused by complex particle image segmentation, this paper proposes a Semi-supervised Hybrid-training Particle Segmentation framework (SHPS) based on skeleton-guided pseudo-labels. First, a pre-trained model is obtained by training a popular segmentation algorithm on partially labeled data. Then, a Background Skeleton-guided Pseudo-label generation algorithm (BSP) is proposed to generate pseudo-labels closer to the ground truth in terms of structural integrity based on coarse segmentation. The final segmentation model is obtained by training a mixed dataset consisting of labeled data and pseudo-labels from another partition on the pre-trained model. The skeleton differences of pseudo-labels and coarse segmentation are added to the loss function. Experimental results show that our method achieves 84.4% accuracy on mIoU with uniform label data distribution, which is 2.1% higher than the accuracy of UNet and reduces the degree of under-segmentation.

Continuous and accurate depth information of blast furnace burden surface is important for optimizing charging operations, thereby reducing its energy consumption and CO2emissions. However, depth estimation for a single image is challenging, especially when estimating the depth of burden surface images in the harsh internal environment of the blast furnace. In this paper, a novel method that is based on edge defocus tracking is proposed to estimate the depth of burden surface images with different morphological characteristics. First, an endoscopic video acquisition system is designed, key frames of burden surface video in stable state are extracted based on feature point optical flow method, and the sparse depth is estimated by using the defocus-based method. Next, the burden surface image is divided into four subregions according to the distribution characteristics of the burden surface, the edge line trajectories and an eight-direction depth gradient template are designed to develop depth propagation rules. Finally, the depth is propagated from edge to the entire image based on edge line tracking method. The experimental results show that the proposed method can accurately and efficiently estimate the depth of the burden surface and provide key data support for optimizing the operation of blast furnace.

Accurate monitoring of abnormalities is of great significance to the stable operation of the blast furnace ironmaking process. This article proposes a data-driven model to accurately monitor the abnormal conditions of blast furnaces. Generally, data-driven models primarily rely on feature extraction from high-dimensional raw data. Recently, deep learning networks have been developed and considered a promising technology in extracting high-level abstract features. However, most of these networks cannot capture deep target-related features for abnormality monitoring. Thus, this article proposes a novel stacked dynamic target-driven denoising autoencoder for layer-by-layer hierarchical feature representation, and the dynamic relationship between samples and targets is described by dynamic factors. Then, we design a corresponding target-driven reconstruction loss function to pretrain the deep network successively. Experimental results in an ironmaking plant demonstrate the effectiveness and feasibility of the proposed method.

An industrial endoscope can monitor the abnormal conditions of the blast furnace burden layer by detecting the burden surface. Aiming at the problem of a small area of burden surface detected by industrial endoscopes due to the harsh environment of high temperature in blast furnaces, a novel optimization method of the installation direction of industrial endoscopes is proposed. First, a modeling method based on the three-dimensional coordinate transformation was proposed to quantitatively describe the relationship between the installation direction and imaged burden surface area. Then, we designed the optimization program based on particle swarm optimization to maximize the imaged burden surface area. Finally, the problems existing in field applications were considered, and the corresponding model of imaged burden surface area was established for the application. Experimental results show that the proposed method can effectively increase the imaged burden surface area, which provides a guarantee for the stable operation of blast furnaces.

An optimal switching heating control strategy based on minimum margin (OS-MM) is proposed for high efficiency, low energy consumption, and high-uniformity fine control in a large-scale vertical quench furnace. This work was stimulated by the need for solving the contradiction between the heating rate and temperature overshoot to achieve a rapid rise in temperature with no overshoot. Based on a three-dimensional (3-D) transient temperature field model of the quenching furnace, the temperature field optimization control problem is transformed into a boundary optimization control problem of rapid heating, whereby it is rigorously proved that the fastest rise in temperature is attained by the full-power heating, which satisfies the bang-bang characteristics. The key parameter that determines the overshoot in the full-power heating mode is introduced and defined as the minimum margin in the temperature-rising process. The analytical expression for solving the minimum margin is calculated by the eigenfunction expansion method and using the strong metal thermal inertia of the internal temperature field, the OS-MM method is designed to stop the heating in advance at a calculated switching point to optimize full-power heating, thus conserving energy by reducing its consumption. The results of simulation and industrial experiments show that this strategy greatly shortens the temperature adjusting time, significantly reduces energy consumption, and effectively improves the control performance indices, such as overshoot and static error in the temperature holding period of the thermal treatment process, as compared to the existing heating methods.

Accurate temperature measurement of blast furnace molten iron is essential for regulating the furnace temperature. However, the dust interference at blast furnace cast field makes it difficult to measure molten iron temperature accurately using the infrared temperature measurement method. To reduce the influence of dust on the results of infrared measurement method, a compensation method using the heterogeneous features of infrared images is proposed in this article. First, the infrared image of the molten iron flow is divided into subregions, from which the subregions containing only molten iron are selected. Then, the statistic and the texture features influenced by dust are extracted from the selected subregions. Finally, the heterogeneous features are used as the compensation model input to estimate the measurement error of each subregion and compensate for the molten iron temperature. Experimental results demonstrate that the compensation method can significantly reduce the infrared measurement errors caused by dust and obtain accurate molten iron temperature.