Lightweight smoke image segmentation is essential for fire warning systems, particularly on mobile devices. In recent years, although numerous high-precision, large-scale smoke segmentation models have been developed, there are few lightweight solutions specifically designed for mobile applications. Therefore, we propose a Multi-stage Group Interaction and Cross-domain Fusion Network (MGICFN) with low computational complexity for real-time smoke segmentation. To improve the model's ability to effectively analyze smoke features, we incorporate a Cross-domain Interaction Attention Module (CIAM) to merge spatial and frequency domain features for creating a lightweight smoke encoder. To alleviate the loss of critical information from small smoke objects during downsampling, we design a Multi-stage Group Interaction Module (MGIM). The MGIM calibrates the information discrepancies between high and low-dimensional features. To enhance the boundary information of smoke targets, we introduce an Edge Enhancement Module (EEM), which utilizes predicted target boundaries as advanced guidance to refine lower-level smoke features. Furthermore, we implement a Group Convolutional Block Attention Module (GCBAM) and a Group Fusion Module (GFM) to connect the encoder and decoder efficiently. Experimental results demonstrate that MGICFN achieves an 88.70% Dice coefficient (Dice), an 81.16% mean Intersection over Union (mIoU), and a 91.93% accuracy (Acc) on the SFS3K dataset. It also achieves an 87.30% Dice, a 78.68% mIoU, and a 92.95% Acc on the SYN70K test dataset. Our MGICFN model has 0.73M parameters and requires 0.3G FLOPs.

Abstract It is very challenging to accurately segment smoke images because smoke has some adverse vision characteristics, such as anomalous shapes, blurred edges, and translucency. Existing methods cannot fully focus on the texture details of anomalous shapes and blurred edges simultaneously. To solve these problems, a Dense Multi‐scale context and Asymmetric pooling Embedding Network (DMAENet) is proposed to model the smoke edge details and anomalous shapes for smoke segmentation. To capture the feature information from different scales, a Dense Multi‐scale Context Module (DMCM) is proposed to further enhance the feature representation capability of our network under the help of asymmetric convolutions. To efficiently extract features for long‐shaped objects, the authors use asymmetric pooling to propose an Asymmetric Pooling Enhancement Module (APEM). The vertical and horizontal pooling methods are responsible for enhancing features of irregular objects. Finally, a Feature Fusion Module (FFM) is designed, which accepts three inputs for improving performance. Low and high‐level features are fused by pixel‐wise summing, and then the summed feature maps are further enhanced in an attention manner. Experimental results on synthetic and real smoke datasets validate that all these modules can improve performance, and the proposed DMAENet obviously outperforms existing state‐of‐the‐art methods.

Abstract Visual smoke semantic segmentation is a challenging task due to semi‐transparency, variable shapes, and complex textures of smoke. To improve segmentation performance, a convolutional neural network and transformer hybrid network are proposed based on pyramid Gaussian pooling (PGP) for smoke segmentation. In order to utilize low‐pass filtering to suppress noise, a PGP method is designed. Then, the output of PGP is reshaped to construct a set of visual tokens for transformers, thus a PGP‐transformer module is presented to make full use of the self‐attention mechanism. Finally, the PGP‐transformer module is inserted into the U‐shaped architecture with skip connections. A large number of experiments have proved that the method is significantly superior to existing state‐of‐the‐art algorithms on virtual and real smoke datasets, and ablation experiments have also verified the effectiveness of the proposed modules.

Aspect-based Sentiment Analysis (ABSA), also known as fine-grained sentiment analysis, aims to predict the sentiment polarity of specific aspect words in the sentence. Some studies have explored the semantic correlation between words in sentences through attention-based methods. Other studies have learned syntactic knowledge by using graph convolution networks to introduce dependency relations. These methods have achieved satisfactory results in the ABSA tasks. However, due to the complexity of language, effectively capturing semantic and syntactic knowledge remains a challenging research question. Therefore, we propose an Adaptive Dual Graph Convolution Fusion Network (AD-GCFN) for aspect-based sentiment analysis. This model uses two graph convolution networks: one for the semantic layer to learn semantic correlations by an attention mechanism, and the other for the syntactic layer to learn syntactic structure by dependency parsing. To reduce the noise caused by the attention mechanism, we designed a module that dynamically updates the graph structure information for adaptively aggregating node information. To effectively fuse semantic and syntactic information, we propose a cross-fusion module that uses the double random similarity matrix to obtain the syntactic features in the semantic space and the semantic features in the syntactic space, respectively. Additionally, we employ two regularizers to further improve the ability to capture semantic correlations. The orthogonal regularizer encourages the semantic layer to learn word semantics without overlap, while the differential regularizer encourages the semantic and syntactic layers to learn different parts. Finally, the experimental results on three benchmark datasets show that the AD-GCFN model is superior to the contrast models in terms of accuracy and macro-F1.

It is quite challenging to visually identify skin lesions with irregular shapes, blurred boundaries and large scale variances. Convolutional Neural Network (CNN) extracts more local features with abundant spatial information, while Transformer has the powerful ability to capture more global information but with insufficient spatial details. To overcome the difficulties in discriminating small or blurred skin lesions, we propose a Bi-directionally Fused Boundary Aware Network (BiFBA-Net). To utilize complementary features produced by CNNs and Transformers, we design a dual-encoding structure. Different from existing dual-encoders, our method designs a Bi-directional Attention Gate (Bi-AG) with two inputs and two outputs for crosswise feature fusion. Our Bi-AG accepts two kinds of features from CNN and Transformer encoders, and two attention gates are designed to generate two attention outputs that are sent back to the two encoders. Thus, we implement adequate exchanging of multi-scale information between CNN and Transformer encoders in a bi-directional and attention way. To perfectly restore feature maps, we propose a progressive decoding structure with boundary aware, containing three decoders with six supervised losses. The first decoder is a CNN network for producing more spatial details. The second one is a Partial Decoder (PD) for aggregating high-level features with more semantics. The last one is a Boundary Aware Decoder (BAD) proposed to progressively improve boundary accuracy. Our BAD uses residual structure and Reverse Attention (RA) at different scales to deeply mine structural and spatial details for refining lesion boundaries. Extensive experiments on public datasets show that our BiFBA-Net achieves higher segmentation accuracy, and has much better ability of boundary perceptions than compared methods. It also alleviates both over-segmentation of small lesions and under-segmentation of large ones.

Compared with other objects, smoke semantic segmentation (SSS) is more difficult and challenging due to some special characteristics of smoke, such as non-rigid, translucency, variable mode and so on. To achieve accurate positioning of smoke in real complex scenes and promote the development of intelligent fire detection, we propose a Smoke-Aware Global-Interactive Non-local Network (SAGINN) for SSS, which harness the power of both convolution and transformer to capture local and global information simultaneously. Non-local is a powerful means for modeling long-range context dependencies, however, friendliness to single-scale low-resolution features limits its potential to produce high-quality representations. Therefore, we propose a Global-Interactive Non-local (GINL) module, leveraging global interaction between multi-scale key information to improve the robustness of feature representations. To solve the interference of smoke-like objects, a Pyramid High-level Semantic Aggregation (PHSA) module is designed, where the learned high-level category semantics from classification aids model by providing additional guidance to correct the wrong information in segmentation representations at the image level and alleviate the inter-class similarity problem. Besides, we further propose a novel loss function, termed Smoke-aware loss (SAL), by assigning different weights to different objects contingent on their importance. We evaluate our SAGINN on extensive synthetic and real data to verify its generalization ability. Experimental results show that SAGINN achieves 83% average mIoU on the three testing datasets (83.33%, 82.72% and 82.94%) of SYN70K with an accuracy improvement of about 0.5%, 0.002 mMse and 0.805Fβ on SMOKE5K, which can obtain more accurate location and finer boundaries of smoke, achieving satisfactory results on smoke-like objects.

Few-shot semantic segmentation is a challenging task that aims to segment novel classes in the query images given only a few annotated support samples. Most existing prototype-based approaches extract global or local prototypes by global average pooling (GAP) or clustering to represent all object information. Subsequently, the prototype information is employed as guidance for query image segmentation. However, these frameworks fail to fully mine the object details and ignore information from query images. Consequently, we propose a Dual-Guided Frequency Prototype Network (DGFPNet) to solve these issues. Specifically, to mine the global and local object information, a Frequency Prototype Generation Module (FPGM) is first proposed to extract more comprehensive frequency prototypes by multi-frequency pooling (MFP) in the DCT domain. Then, with the guidance of support and query information, a Dual-Guided Selection Module (DGSM) is presented to produce the query attention mask and select more effective prototypes. Based on the query attention mask and support information, the generalized object information is integrated into the feature with the proposed Feature Generalization Module (FGM). Finally, we propose a Multi-Dimension Feature Enrichment Decoder Module (MDFEDM) to capture multi-dimension object information and tackle hard pixels for refining the final segmentation results. Extensive experiments on PASCAL-5iand COCO-20ishow that our model achieves new state-of-the-art performances. Our code will be released athttps://github.com/ChunLinWen/DGFPNet.

Abstract Segmentations provide important clues for diagnosing diseases. U‐shaped neural networks with skip connections have become one of popular frameworks for medical image segmentation. Skip connections really reduce loss of spatial details caused by down‐sampling, but they cannot handle well semantic gaps between low‐ and high‐level features. It is quite challenging to accurately separate out long, narrow, and small organs from human bodies. To solve these problems, the authors propose a Multiple Gated Boosting Network (MGB‐Net). To boost spatial accuracy, the authors first adopt Gated Recurrent Units (GRU) to design multiple Gated Skip Connections (GSC) at different levels, which efficiently reduce the semantic gap between the shallow and deep features. The Update and Reset gates of GRUs enhance features beneficial to segmentation and suppress information adverse to final results in a recurrent way. To obtain more scale invariances, the authors propose a module of Multi‐scale Weighted Channel Attention (MWCA). The module first uses convolutions with different kernel sizes and group numbers to generate multi‐scale features, and then adopts learnable weights to emphasize the importance of each scale for capturing attention features. Blocks of Transformer Self‐Attention (TSA) are sequentially stacked to extract long‐range dependency features. To effectively fuse and boost the features of MWCA and TSA, the authors use GRUs again to propose a Gated Dual Attention module (GDA), which enhances beneficial features and suppresses adverse information in a gated learning way. Experiments show that the authors’ method achieves an average Dice coefficient of 80.66% on the Synapse multi‐organ segmentation dataset. The authors’ method outperforms the state‐of‐the‐art methods on medical images. In addition, the authors’ method achieves a Dice segmentation accuracy of 62.77% on difficult objects such as pancreas, significantly exceeding the current average accuracy, so multiple gated boosting (MGB) methods are reliably effective for improving the ability of feature representations. The authors’ code is publicly available at https://github.com/DAgalaxy/MGB‐Net .

Smoke has semi-transparency property leading to highly complicated mixture of background and smoke. Sparse or small smoke is visually inconspicuous, and its boundary is often ambiguous. These reasons result in a very challenging task of separating smoke from a single image. To solve these problems, we propose a Classification-assisted Gated Recurrent Network (CGRNet) for smoke semantic segmentation. To discriminate smoke and smoke-like objects, we present a smoke segmentation strategy with dual classification assistance. Our classification module outputs two prediction probabilities for smoke. The first assistance is to use one probability to explicitly regulate the segmentation module for accuracy improvement by supervising a cross-entropy classification loss. The second one is to multiply the segmentation result by another probability for further refinement. This dual classification assistance greatly improves performance at image level. In the segmentation module, we design an Attention Convolutional GRU module (Att-ConvGRU) to learn the long-range context dependence of features. To perceive small or inconspicuous smoke, we design a Multi-scale Context Contrasted Local Feature structure (MCCL) and a Dense Pyramid Pooling Module (DPPM) for improving the representation ability of our network. Extensive experiments validate that our method significantly outperforms existing state-of-art algorithms on smoke datasets, and also obtain satisfactory results on challenging images with inconspicuous smoke and smoke-like objects.

Smoke density estimation from a single image is a totally new but highly ill-posed problem. To solve the problem, we stack several convolutional encoder-decoder structures together to propose a wave-shaped neural network, termed W-Net. Stacking encoder-decoders directly increases the network depth, leading to the enlargement of receptive fields for encoding more semantic information. To maximize the degrees of feature re-usage, we copy and resize the outputs of encoding layers to corresponding decoding layers, and then concatenate them to implement short-cut connections for improving spatial accuracy. The crests and troughs of W-Net are special structures containing abundant localization and semantic information, so we also use short-cut connections between these structures and decoding layers. Estimated smoke density is useful in many applications, such as smoke segmentation, smoke detection, disaster simulation. Experimental results show that our method outperforms existing methods on both smoke density estimation and segmentation. It also achieves satisfying results in visual detection of auto exhausts.

Atypical Hepatocellular Carcinoma (HCC) is very hard to distinguish from Focal Nodular Hyperplasia (FNH) in routine imaging. However little attention was paid to this problem. This paper proposes a novel liver tumor Computer-Aided Diagnostic (CAD) approach extracting spatio-temporal semantics for atypical HCC. With respect to useful diagnostic semantics, our model automatically calculates three types of semantic feature with equally down-sampled frames based on Contrast-Enhanced Ultrasound (CEUS). Thereafter, a Support Vector Machine (SVM) classifier is trained to make the final diagnosis. Compared with traditional methods for diagnosing HCC, the proposed model has the advantage of less computational complexity and being able to handle the atypical HCC cases. The experimental results show that our method obtained a pretty considerable performance and outperformed two traditional methods. According to the results, the average accuracy reaches 94.40%, recall rate 94.76%, F1-score value 94.62%, specificity 93.62% and sensitivity 94.76%, indicating good merit for automatically diagnosing atypical HCC cases.

It is challenging to recognize smoke from visual scenes due to large variations of smoke colors, textures and shapes. To improve robustness, we propose a novel feature extraction method based on similarity and dissimilarity matching measures of Local Binary Patterns (LBP). Given two bit‐sequences of an LBP code pair, the similarity and dissimilarity matching measures are defined as the ratios of the 1–1 bitwise matching number to the 0–0 bitwise matching number and the 1–0 number to the 0–1 number, respectively. To capture local code variations, we calculate the measures between LBP codes of a center pixel and its neighbors. Then we compare each measure with its global mean to propose Similarity Matching based Local Binary Patterns (SMLBP) and Dissimilarity Matching based Local Binary Patterns (DMLBP). Since SMLBP and DMLBP extract spatial variations of the 1st order LBP codes, they actually represent the 2nd order variations of pixel values. Furthermore, we adopt different mapping modes and multi‐scale neighborhoods to obtain rotation and scale invariances. Finally, we concatenate the histograms of LBP, SMLBP and DMLBP to generate a feature vector containing 1st and 2nd order information. Experiments show that our method obviously outperforms existing methods.

To improve recognition accuracy, the authors fuse texture, edge and line information to propose a feature extraction method for smoke recognition. The Canny operator is proposed to generate an edge image from an original image, and then adopt the Hough transform to extract straight lines from the edge image. The lines are rasterised to generate a discrete line image and two local patterns are proposed for the edge and line images. The first one is local boundary summation pattern (LBSP) that computes the sum of binary pixel values along the boundary of a local region around a centre pixel. The second one is called local region summation pattern (LRSP) that sums up the binary values of pixels in a local region around the centre pixel. Besides LBSP and LRSP, LBPs with three mapping modes (LBP_M3) to achieve traditional texture information are also extracted. Finally, the authors concatenate the histograms of LBP_M3, LBSP and LRSP to generate a feature vector, and use support vector machine for classifying and testing. Experiments show that authors’ method outperforms most of existing traditional methods for smoke recognition. Although this method has low dimensional features, it also obtains good performance for multi‐class texture classification.

With various emerging mobile devices, the visual content have be to resized into different sizes or aspect ratios for good viewing experiences. In this paper, we propose a new multioperator retargeting algorithm by using four retargeting operators of seam carving, cropping, warping, and scaling iteratively. To determine which retargeting operator should be used at each iteration, we adopt structural similarity (SSIM) to evaluate the similarity between the original and retargeted images. The retargeting operator sequence is constructed based on the four types of retargeting operators by an optimization process. Since the sizes of original and retargeted images are different, scale-invariant feature transform flow is used for dense correspondence between the original and retargeted images for similarity evaluation. Additionally, visual saliency is used to weight SSIM results based on the characteristics of the human visual system. Experimental results on a public image retargeting database have shown the promising performance of the proposed multioperator retargeting algorithm.

It is very challenging to accurately detect smoke from images because of large variances of smoke colour, textures, shapes and occlusions. To improve performance, the authors combine dual threshold AdaBoost with staircase searching technique to propose and implement an image smoke detection method. First, extended Haar‐like features and statistical features are efficiently extracted from integral images from both intensity and saturation components of RGB images. Then, a dual threshold AdaBoost algorithm with a staircase searching technique is proposed to classify the features of smoke for smoke detection. The staircase searching technique aims at keeping consistency of training and classifying as far as possible. Finally, dynamic analysis is proposed to further validate the existence of smoke. Experimental results demonstrate that the proposed system has a good robustness in terms of early smoke detection and low false alarm rate, and it can detect smoke from videos with size of 320 × 240 in real time.

An efficient 3D visualization system has not only fast volume rendering algorithms but also effective navigation methods. Rendering speed is one of key technologies in most 3D visualization applications. We exploit software, Pentium 4 and graphics hardware technologies, such as threshold segmentation, Intel SIMD and 3D texturing, to obtain interactive volume rendering on a standard PC without specialized expensive hardware. Path planning is essential in many 3D visualization applications, such as virtual endocopy, in order to accelerate exploring. There are three major types of methods to extract the navigation path from a 3D data set, including manual, 3D distance transform and thinning based techniques. 3D thinning is a desirable method to extract skeletons of objects, but it has some severe problems to be solved. It is time consuming with discontinuity and small branches. An effective encoding and coordinates transform based scheme is presented to generate a look up table of 3D thinning templates to speed up path extracting, and a two-pass tracking technique is followed to trim the small branches of skeletons. Tri-pass cubic Bezier technique is proposed to decrease large curvatures caused by discrete representation of path. A smooth and C 1 continuity navigation path is thus produced by our algorithms. Following this path, the camera moves and rotates smoothly without any dithering. Our system is very useful and can be widely applied due to full utilization of the existing inexpensive capabilities of PCs.