Aspect-Term Sentiment Analysis (ATSA) in multi-aspect sentences faces a fundamental tradeoff between efficiency and expressiveness.Existing models either re-encode the sentence for each aspect or rely on static use of deep representations, leading to redundant computation and limited adaptivity.We argue that Transformer depth is a costly, queryable resource, and propose DABS, a single-pass inference framework that encodes each sentence once to construct a reusable, depth-ordered substrate.Each aspect then queries this shared representation to selectively read relevant tokens and abstraction levels, without re-encoding.This decouples shared sentence encoding from lightweight, aspect-conditioned readout.Experiments on four ATSA benchmarks show that DABS achieves competitive performance while reducing end-to-end computation by up to 60% in multi-aspect settings (M ≥ 2).Further analyses indicate that adaptive depth querying is most beneficial for linguistically complex cases such as negation and contrast.

Generative AI presents transformative potential across various domains, from creative arts to scientific visualization. However, the utility of AI-generated imagery is often compromised by visual flaws, including anatomical inaccuracies, improper object placements, and misplaced textual elements. These imperfections pose significant challenges for practical applications. To overcome these limitations, we introduce Yuan, a novel framework that autonomously corrects visual imperfections in text-to-image synthesis. Yuan uniquely conditions on both the textual prompt and the segmented image, generating precise masks that identify areas in need of refinement without requiring manual intervention—a common constraint in previous methodologies. Following the automated masking process, an advanced inpainting module seamlessly integrates contextually coherent content into the identified regions, preserving the integrity and fidelity of the original image and associated text prompts. Through extensive experimentation on publicly available datasets such as ImageNet100 and Stanford Dogs, along with a custom-generated dataset, Yuan demonstrated superior performance in eliminating visual imperfections. Our approach consistently achieved higher scores in quantitative metrics, including NIQE, BRISQUE, and PI, alongside favorable qualitative evaluations. These results underscore Yuan's potential to significantly enhance the quality and applicability of AI-generated images across diverse fields.

We present TCM-VR, a domain-specific multimodal large language model (MLLM) for Traditional Chinese Medicine (TCM) capable of recognizing images of dried herbs and answering clinical-type multiple-choice questions (MCQs). Trained and finedtuned upon 220 K images of herbs in 163 classes and 880 K candidate answers for MCQs, respectively, based on Qwen2.5VL, TCM-VR bridges the gap between visual recognition and symbolic reasoning in TCM education and diagnostics. In an effort to decrease overfitting and respond-position bias, we present a Cross-Transformation Memory Mechanism (CTMM) which enforces semantic coherence through paraphrasing prompts and shuffled answer orders at training time. This enhances the model's generalizability and robustness to variations in input. TCMVR achieves 96.7 % MCQ accuracy in held-out test suites and significantly outperforms general-purpose models like GPT-4o and Gemini. Case analyses confirm the model reason beyond content rather than memorizing answer locations. It shows favorable potential for using TCM-VR in learning materials, clinical decision support systems, and intelligent herb dispensary systems, which constitutes advancements in interpretable, multimodal traditional medicine AI. Our model is available at https://huggingface.co/dylanyang963/TCM-VR.

Federated Foundation Models (FedFMs) represent a distributed learning paradigm that fuses general competences of foundation models as well as privacy-preserving capabilities of federated learning. This combination allows the large foundation models and the small local domain models at the remote clients to learn from each other in a teacher-student learning setting. This paper provides a comprehensive summary of the ten challenging problems inherent in FedFMs, encompassing foundational theory, utilization of private data, continual learning, unlearning, Non-IID and graph data, bidirectional knowledge transfer, incentive mechanism design, game mechanism design, model watermarking, and efficiency. The ten challenging problems manifest in five pivotal aspects: “Foundational Theory,” which aims to establish a coherent and unifying theoretical framework for FedFMs. “Data,” addressing the difficulties in leveraging domain-specific knowledge from private data while maintaining privacy; “Heterogeneity,” examining variations in data, model, and computational resources across clients; “Security and Privacy,” focusing on defenses against malicious attacks and model theft; and “Efficiency,” highlighting the need for improvements in training, communication, and parameter efficiency. For each problem, we offer a clear mathematical definition on the objective function, analyze existing methods, and discuss the key challenges and potential solutions. This in-depth exploration aims to advance the theoretical foundations of FedFMs, guide practical implementations, and inspire future research to overcome these obstacles, thereby enabling the robust, efficient, and privacy-preserving FedFMs in various real-world applications.

Large-scale text-to-image (T2I) diffusion models have showcased incredible capabilities in generating coherent images based on textual descriptions, enabling vast applications in content generation. While recent advancements have introduced control over factors such as object localization, posture, and image contours, a crucial gap remains in our ability to control the interactions between objects in the generated content. Well-controlling interactions in generated images could yield meaningful applications, such as creating realistic scenes with interacting characters. In this work, we study the problems of conditioning T2I diffusion models with Human-Object Interaction (HOI) information, consisting of a triplet label (person, action, object) and corresponding bounding boxes. We propose a pluggable interaction control model, called InteractDiffusion that extends existing pre-trained T2I diffusion models to enable them being better conditioned on interactions. Specifically, we tokenize the HOI information and learn their relationships via interaction embeddings. A conditioning self-attention layer is trained to map HOI tokens to visual tokens, thereby conditioning the visual tokens better in existing T2I diffusion models. Our model attains the ability to control the interaction and location on existing T2I diffusion models, which outperforms existing baselines by a large margin in HOI detection score, as well as fidelity in FID and KID. Project page: https://jiuntian.github.io/interactdiffusion.

The advent of Federated Learning (FL) highlights the practical necessity for the ’right to be forgotten’ for all clients, allowing them to request data deletion from the machine learning model’s service provider. This necessity has spurred a growing demand for Federated Unlearning (FU). Feature unlearning has gained considerable attention due to its applications in unlearning sensitive, backdoor, and biased features. Existing methods employ the influence function to achieve feature unlearning, which is impractical for FL as it necessitates the participation of other clients, if not all, in the unlearning process. Furthermore, current research lacks an evaluation of the effectiveness of feature unlearning. To address these limitations, we define feature sensitivity in evaluating feature unlearning according to Lipschitz continuity. This metric characterizes the model output’s rate of change or sensitivity to perturbations in the input feature. We then propose an effective federated feature unlearning framework called Ferrari, which minimizes feature sensitivity. Extensive experimental results and theoretical analysis demonstrate the effectiveness of Ferrari across various feature unlearning scenarios, including sensitive, backdoor, and biased features. The code is publicly available at https://github.com/OngWinKent/Federated-Feature-Unlearning

Recent advancements in Scene Text Visual Question Answering (Text-VQA) employ autoregressive Transformers, showing improved performance with larger models and pre-training datasets. Although various pruning frameworks exist to simplify Transformers, many are integrated into the time-consuming training process. Researchers have recently explored post-training pruning techniques, which separate pruning from training and reduce time consumption. Some methods use gradient-based importance scores that rely on labeled data, while others offer retraining-free algorithms that quickly enhance pruned model accuracy. This paper proposes a novel gradient-based importance score that only necessitates raw, unlabeled data for post-training structured autoregressive Transformer pruning. Additionally, we introduce a Retraining Strategy (ReSt) for efficient performance restoration of pruned models of arbitrary sizes. We evaluate our approach on TextVQA and ST-VQA datasets using TAP, TAP †† and SaL‡-Base where all utilize autoregressive Transformers. On TAP and TAP †† , our pruning approach achieves up to 60% reduction in size with less than a 2.4% accuracy drop and the proposed ReSt retraining approach takes only 3 to 34 min, comparable to existing retraining-free techniques. On SaL‡-Base , the proposed method achieves up to 50% parameter reduction with less than 2.9% accuracy drop requiring only 1.19 h of retraining using the proposed ReSt approach. The code is publicly accessible at https://github.com/soonchangAI/LFPR .

Deep learning models trained on long-tailed (LT) datasets often exhibit bias towards head classes with high frequency. This paper highlights the limitations of existing solutions that combine class- and instance-level re-weighting loss in a naive manner. Specifically, we demonstrate that such solutions result in overfitting the training set, significantly impacting the rare classes. To address this issue, we propose a novel loss function that dynamically reduces the influence of outliers and assigns class-dependent focusing parameters. We also introduce a new long-tailed dataset, ICText-LT, featuring various image qualities and greater realism than artificially sampled datasets. Our method has proven effective, outperforming existing methods through superior quantitative results on CIFAR-LT, Tiny ImageNet-LT, and our new ICText-LT datasets. The source code and new dataset are available at https://github.com/nwjun/FFDS-Loss.

Limited generalization capability has been an unsolved issue in the domain of low-light image enhancement. Many models find enhancing out-of-distribution underexposed images challenging. In this work, we offer a fresh point of view on this issue. Our approach involves dividing the enhancement process into many small steps and performing them gradually. This method allows the model to acquire a more robust understanding of the data. To put this concept into practice, we proposed to adopt a diffusion model for low-light image enhancement, as its way of encoding the mapping between the source and target distributions fits our idea. Empirically, we show that our proposed model (LLDE) can outperform recent SOTAs quantitatively and visually. The code is publicly available at https://github.com/OoiXinPeng/LLDE.

A difficulty of global-level translation is to preserve instance-level details in an image. Although some instance level translation methods can retain the details, most of them require either pre-trained object detection/segmentation network or annotation labels. In this work, we propose a novel method namely CyEDA to perform global level domain adaptation that can preserve image contents without any pre-trained networks integration or annotation labels. Specifically, we introduce blending masks and cycle-object edge consistency loss which exploit the preservation of image objects. We show that our approach can outperform other SOTAs in terms of image quality and FID score in both BDD100K and GTA datasets. The code and pre-trained models are publicly available at https://github.com/bjc1999/CyEDA.

The usage of edge models in medical field has a huge impact on promoting the accessibility of real-time medical services in the under-developed regions. However, the handling of latency-accuracy trade-off to produce such an edge model is very challenging. Although the recent Once-For-All (OFA) network is able to directly produce a set of sub-network designs with Progressive Shrinking (PS) algorithm, it still suffers from training resource and time inefficiency downfall. In this paper, we propose a new OFA training algorithm, namely the Progressive Expansion (ProX). Empirically, we showed that the proposed paradigm can reduce training time up to 68%; while still able to produce sub-networks that have either similar or better accuracy compared to those trained with OFA-PS in ROCT (classification), BRATS and Hippocampus (3D-segmentation) public medical datasets.

Deep learning-based methods have made impressive progress in enhancing extremely low-light images - the image quality of the reconstructed images has generally improved. However, we found out that most of these methods could not sufficiently recover the image details, for instance, the texts in the scene. In this paper, a novel image enhancement framework is proposed to precisely restore the scene texts, as well as the overall quality of the image simultaneously under extremely low-light conditions. Mainly, we employed a self-regularised attention map, an edge map, and a novel text detection loss. In addition, leveraging the synthetic low-light images is beneficial for image enhancement on the genuine ones in terms of text detection. The quantitative and qualitative experimental results have shown that the proposed model outperforms state-of-the-art methods in image restoration, text detection, and text spotting on See In the Dark and ICDAR15 datasets.

With substantial amount of time, resources and human (team) efforts invested to explore and develop successful deep neural networks (DNN), there emerges an urgent need to protect these inventions from being illegally copied, redistributed, or abused without respecting the intellectual properties of legitimate owners. Following recent progresses along this line, we investigate a number of watermark-based DNN ownership verification methods in the face of ambiguity attacks, which aim to cast doubts on the ownership verification by forging counterfeit watermarks. It is shown that ambiguity attacks pose serious threats to existing DNN watermarking methods. As remedies to the above-mentioned loophole, this paper proposes novel passport-based DNN ownership verification schemes which are both robust to network modifications and resilient to ambiguity attacks. The gist of embedding digital passports is to design and train DNN models in a way such that, the DNN inference performance of an original task will be significantly deteriorated due to forged passports. In other words, genuine passports are not only verified by looking for the predefined signatures, but also reasserted by the unyielding DNN model inference performances. Extensive experimental results justify the effectiveness of the proposed passport-based DNN ownership verification schemes. Code is available at https://github.com/kamwoh/DeepIPR.

Ever since Machine Learning as a Service emerges as a viable business that utilizes deep learning models to generate lucrative revenue, Intellectual Property Right (IPR) has become a major concern because these deep learning models can easily be replicated, shared, and re-distributed by any unauthorized third parties. To the best of our knowledge, one of the prominent deep learning models - Generative Adversarial Networks (GANs) which has been widely used to create photorealistic image are totally unprotected despite the existence of pioneering IPR protection methodology for Convolutional Neural Networks (CNNs). This paper therefore presents a complete protection framework in both black-box and white-box settings to enforce IPR protection on GANs. Empirically, we show that the proposed method does not compromise the original GANs performance (i.e. image generation, image super-resolution, style transfer), and at the same time, it is able to withstand both removal and ambiguity attacks against embedded watermarks. Codes are available at https://github.com/dingsheng-ong/ipr-gan.

Deep neural networks (DNN) are widely used in real-life applications despite the lack of understanding on this technology and its challenges. Data privacy is one of the bottlenecks that is yet to be overcome and more challenges in DNN arise when researchers start to pay more attention to DNN vulnerabilities. In this work, we aim to cast the doubts towards the reliability of the DNN with solid evidence particularly in Federated Learning environment by utilizing an existing privacy breaking algorithm which inverts gradients of models to reconstruct the input data. By performing the attack algorithm, we exemplify the data reconstructed from inverting gradients algorithm as a potential threat and further reveal the vulnerabilities of models in representation learning. Pytorch implementation are provided at https://github.com/Jiaqi0602/adversarial-attack-from-leakage/

A deep hashing model typically has two main learning objectives: to make the learned binary hash codes discriminative and to minimize a quantization error. With further constraints such as bit balance and code orthogonality, it is not uncommon for existing models to employ a large number (>4) of losses. This leads to difficulties in model training and subsequently impedes their effectiveness. In this work, we propose a novel deep hashing model with only $\textit{a single learning objective}$. Specifically, we show that maximizing the cosine similarity between the continuous codes and their corresponding $\textit{binary orthogonal codes}$ can ensure both hash code discriminativeness and quantization error minimization. Further, with this learning objective, code balancing can be achieved by simply using a Batch Normalization (BN) layer and multi-label classification is also straightforward with label smoothing. The result is a one-loss deep hashing model that removes all the hassles of tuning the weights of various losses. Importantly, extensive experiments show that our model is highly effective, outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins.

Hospital readmission is a major cost for healthcare systems worldwide. If patients with a higher potential of readmission could be identified at the start, existing resources could be used more efficiently, and appropriate plans could be implemented to reduce the risk of readmission. Therefore, it is important to predict the right target patients. Medical data is usually noisy, incomplete, and inconsistent. Hence, before developing a prediction model, it is crucial to efficiently set up the predictive model so that improved predictive performance is achieved. The current study aims to analyse the impact of different preprocessing methods on the performance of different machine learning classifiers. The preprocessing applied by previous hospital readmission studies were compared, and the most common approaches highlighted such as missing value imputation, feature selection, data balancing, and feature scaling. The hyperparameters were selected using Bayesian optimisation. The different preprocessing pipelines were assessed using various performance metrics and computational costs. The results indicated that the preprocessing approaches helped improve the model’s prediction of hospital readmission.

Visual Question Answering (VQA) methods have made incredible progress, but suffer from a failure to generalize. This is visible in the fact that they are vulnerable to learning coincidental correlations in the data rather than deeper relations between image content and ideas expressed in language. We present a dataset that takes a step towards addressing this problem in that it contains questions expressed in two languages, and an evaluation process that co-opts a well understood image-based metric to reflect the method’s ability to reason. Measuring reasoning directly encourages generalization by penalizing answers that are coincidentally correct. The dataset reflects the scene-text version of the VQA problem, and the reasoning evaluation can be seen as a text-based version of a referring expression challenge. Experiments and analyses are provided that show the value of the dataset. The dataset is available at www.est-vqa.org.

Recent works have shown success in generating music using a Variational Autoencoder (VAE). However, we found out that the style of the generated music is usually governed or limited by the training dataset. In this work, we proposed a new formulation to the VAE that allows users to condition on the style of the generated music. Technically, our VAE consists of two latent spaces- content and style space to encode the content and style of a song separately. Each style is represented by a continuous style embedding, unlike previous works which mostly used discrete or one-hot style labels. We trained our model on public datasets that made up of Bach chorales and western folk tunes. Empirically, as well as from music theory point of view, we show that our proposed model can generate better music samples of each style than a baseline model. The source code and generated samples are available at https: //git;hub. com/daQuincy /DeepMu sicvSt; yle.

Early hierarchical computational visual models as well as recent deep neural networks have been inspired by the functioning of the primate visual cortex system. Although much effort has been made to dissect neural networks to visualize the features they learn at the individual units, the scope of the visualizations has been limited to a categorization of the features in terms of their semantic level. Considering the ability humans have to select high semantic level regions of a scene, the question whether neural networks can match this ability, and if similarity with humans attention is correlated with neural networks performance naturally arise. To address this question we propose a pipeline to select and compare sets of feature points that maximally activate individual networks units to human fixations. We extract features from a variety of neural networks, from early hierarchical models such as HMAX up to recent deep convolutional neural netwoks such as Densnet, to compare them to human fixations. Experiments over the ETD database show that human fixations correlate with CNNs features from deep layers significantly better than with random sets of points, while they do not with features extracted from the first layers of CNNs, nor with the HMAX features, which seem to have low semantic level compared with the features that respond to the automatically learned filters of CNNs. It also turns out that there is a correlation between CNN's human similarity and classification performance.

This paper proposes a novel deep polarized network (DPN) for learning to hash, in which each channel in the network outputs is pushed far away from zero by employing a differentiable bit-wise hinge-like loss which is dubbed as polarization loss. Reformulated within a generic Hamming Distance Metric Learning framework [Norouzi et al., 2012], the proposed polarization loss bypasses the requirement to prepare pairwise labels for (dis-)similar items and, yet, the proposed loss strictly bounds from above the pairwise Hamming Distance based losses. The intrinsic connection between pairwise and pointwise label information, as disclosed in this paper, brings about the following methodological improvements: (a) we may directly employ the proposed differentiable polarization loss with no large deviations incurred from the target Hamming distance based loss; and (b) the subtask of assigning binary codes becomes extremely simple --- even random codes assigned to each class suffice to result in state-of-the-art performances, as demonstrated in CIFAR10, NUS-WIDE and ImageNet100 datasets.

This paper reports the ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text - RRC-ArT that consists of three major challenges: i) scene text detection, ii) scene text recognition, and iii) scene text spotting. A total of 78 submissions from 46 unique teams/individuals were received for this competition. The top performing score of each challenge is as follows: i) T1 - 82.65%, ii) T2.1 - 74.3%, iii) T2.2 - 85.32%, iv) T3.1 - 53.86%, and v) T3.2 - 54.91%. Apart from the results, this paper also details the ArT dataset, tasks description, evaluation metrics and participants' methods. The dataset, the evaluation kit as well as the results are publicly available at the challenge website.

Robust text reading from street view images provides valuable information for various applications. Performance improvement of existing methods in such a challenging scenario heavily relies on the amount of fully annotated training data, which is costly and in-efficient to obtain. To scale up the amount of training data while keeping the labeling procedure cost-effective, this competition introduces a new challenge on Large-scale Street View Text with Partial Labeling (LSVT), providing 5,0000 and 400,000 images in full and weak annotations, respectively. This competition aims to explore the abilities of state-of-the-art methods to detect and recognize text instances from large-scale street view images, closing gaps between research benchmarks and real applications. During the competition period, a total number of 41 teams participate in the two tasks with 132 valid submissions, i.e., text detection and end-to-end text spotting. This paper includes dataset descriptions, task definitions, evaluation protocols and results summaries of ICDAR 2019-LSVT challenge.

Nowadays, attention mechanisms have been widely adopted in image captioning task due to its outstanding performance. In this paper, we propose a Mask Captioning Network (MaC) that consists of an object layer and a background layer to capture the objects and scene of an image, independently to generate a much richer sentence. To this end, we leverage on Mask RCNN to detect the salient regions in pixel level in the object layer; while, in the background layer, a CNN model is used to encode the scene features. Experimental results show that our model significantly outperforms baseline models and achieves comparable results with the state-of-the-art methods on MSCOCO and Flickr30k datasets.

Collective behaviour of coherent groups conveys the semantic relations among individuals in a crowd scene. However, classically, crowd analysis in still image is either focused on crowd counting estimation or crowd segmentation only. In this paper, we present a novel framework that merges these two classical approaches together as one to achieve a higher level crowd understanding, i.e. detecting coherent groups within a crowd in still image. Essentially, in addition to crowd counting estimation, our work is able to infer crowd segments at both image level and coherent group level. Experimental results on ShanghaiTech dataset showed the efficacy of our solution.

With substantial amount of time, resources and human (team) efforts invested to explore and develop successful deep neural networks (DNN), there emerges an urgent need to protect these inventions from being illegally copied, redistributed, or abused without respecting the intellectual properties of legitimate owners. Following recent progresses along this line, we investigate a number of watermark-based DNN ownership verification methods in the face of ambiguity attacks, which aim to cast doubts on the ownership verification by forging counterfeit watermarks. It is shown that ambiguity attacks pose serious threats to existing DNN watermarking methods. As remedies to the above-mentioned loophole, this paper proposes novel passport-based DNN ownership verification schemes which are both robust to network modifications and resilient to ambiguity attacks. The gist of embedding digital passports is to design and train DNN models in a way such that, the DNN inference performance of an original task will be significantly deteriorated due to forged passports. In other words, genuine passports are not only verified by looking for the predefined signatures, but also reasserted by the unyielding DNN model inference performances. Extensive experimental results justify the effectiveness of the proposed passport-based DNN ownership verification schemes. Code and models are available at https://github.com/kamwoh/DeepIPR

This paper proposes a series of new approaches to improve Generative Adversarial Network (GAN) for conditional image synthesis and we name the proposed model as "ArtGAN". One of the key innovation of ArtGAN is that, the gradient of the loss function w.r.t. the label (randomly assigned to each generated image) is back-propagated from the categorical discriminator to the generator. With the feedback from the label information, the generator is able to learn more efficiently and generate image with better quality. Inspired by recent works, an autoencoder is incorporated into the categorical discriminator for additional complementary information. Last but not least, we introduce a novel strategy to improve the image quality. In the experiments, we evaluate ArtGAN on CIFAR-10 and STL-10 via ablation studies. The empirical results showed that our proposed model outperforms the state-of-the-art results on CIFAR-10 in terms of Inception score. Qualitatively, we demonstrate that ArtGAN is able to generate plausible-looking images on Oxford-102 and CUB-200, as well as able to draw realistic artworks based on style, artist, and genre. The source code and models are available at: https://github.com/cs-chan/ArtGAN.

Recent works in image captioning have shown very promising raw performance. However, we realize that most of these encoder-decoder style networks with attention do not scale naturally to large vocabulary size, making them difficult to deploy on embedded systems with limited hardware resources. This is because the size of word and output embedding matrices grow proportionally with the size of vocabulary, adversely affecting the compactness of these networks. To address this limitation, this paper introduces a brand new idea in the domain of image captioning. That is, we tackle the problem of compactness of image captioning models which is hitherto unexplored. We showed that our proposed model, named COMIC for compact image captioning, achieves comparable results in five common evaluation metrics with state-of-the-art approaches on both MS-COCO and InstaPIC-1.1M datasets despite having an embedded vocabulary size that is 39×-99× smaller.

In this paper, we propose a novel video saliency detection method using the Partial Differential Equations (PDEs). We first form a static adaptive anisotropic PDE model from the unpredicted frames of the video using a detection map and a saliency seeds set of most attractive image elements. At the same time, we also extract motion features from the predicted frames of the video to generate motion saliency map. Then, we combine these two maps to obtain the final saliency map (video). Experiments on various human-action datasets show that our video saliency detection model performs favourably against the conventional solutions.

How does a pre-trained Convolution Neural Network (CNN) model perform on beauty and personal care items (i.e Perfect-500K) This is the question we attempt to answer in this paper by adopting several well known deep learning models pre-trained on ImageNet, and evaluate their performance using different distance metrics. In the Perfect Corp Challenge, we manage to secure fourth position by using only the pre-trained model.

Classification of plants based on a multi-organ approach is very challenging. Although additional data provide more information that might help to disambiguate between species, the variability in shape and appearance in plant organs also raises the degree of complexity of the problem. Despite promising solutions built using deep learning enable representative features to be learned for plant images, the existing approaches focus mainly on generic features for species classification, disregarding the features representing plant organs. In fact, plants are complex living organisms sustained by a number of organ systems. In our approach, we introduce a hybrid generic-organ convolutional neural network (HGO-CNN), which takes into account both organ and generic information, combining them using a new feature fusion scheme for species classification. Next, instead of using a CNN-based method to operate on one image with a single organ, we extend our approach. We propose a new framework for plant structural learning using the recurrent neural network-based method. This novel approach supports classification based on a varying number of plant views, capturing one or more organs of a plant, by optimizing the contextual dependencies between them. We also present the qualitative results of our proposed models based on feature visualization techniques and show that the outcomes of visualizations depict our hypothesis and expectation. Finally, we show that by leveraging and combining the aforementioned techniques, our best network outperforms the state of the art on the PlantClef2015 benchmark. The source code and models are available at https://github.com/cs-chan/Deep-Plant.

Text in curve orientation, despite being one of the common text orientations in real world environment, has close to zero existence in well received scene text datasets such as ICDAR'13 and MSRA-TD500. The main motivation of Total-Text is to fill this gap and facilitate a new research direction for the scene text community. On top of conventional horizontal and multi-oriented text, it features curved-oriented text. Total-Text is highly diversified in orientations, more than half of its images have a combination of more than two orientations. Recently, a new breed of solutions that casted text detection as a segmentation problem has demonstrated their effectiveness against multi-oriented text. In order to evaluate its robustness against curved text, we fine-tuned DeconvNet and benchmark it on Total-Text. Total-Text with its annotation is available at https://github.com/cs-chan/Total-Text-Dataset.

Classification of plants based on a multi-organ approach is very challenging. Although additional data provides more information that might help to disambiguate between species, the variability in shape and appearance in plant organs also raises the degree of complexity of the problem. Existing approaches focus mainly on generic features for species classification, disregarding the features representing the organs. In fact, plants are complex entities sustained by a number of organ systems. In our approach, we exploit the PlantClef2015 benchmark, and introduce a hybrid generic-organ convolutional neural network (HGO-CNN), which takes into account both organ and generic information, combining them using a new feature fusion scheme for species classification. We show that our proposed method outperforms the state-of-the-art results.

This paper proposes an extension to the Generative Adversarial Networks (GANs), namely as ArtGAN to synthetically generate more challenging and complex images such as artwork that have abstract characteristics. This is in contrast to most of the current solutions that focused on generating natural images such as room interiors, birds, flowers and faces. The key innovation of our work is to allow back-propagation of the loss function w.r.t. the labels (randomly assigned to each generated images) to the generator from the discriminator. With the feedback from the label information, the generator is able to learn faster and achieve better generated image quality. Empirically, we show that the proposed ArtGAN is capable to create realistic artwork, as well as generate compelling real world images that globally look natural with clear shape on CIFAR-10.

Crowd segmentation is important in serving as the basis for a wide range of crowd analysis tasks such as density estimation and behavior understanding. However, due to interocclusions, perspective distortion, clutter background, and random crowd distribution, localizing crowd segments is technically a very challenging task. This paper proposes a novel crowd segmentation framework-based on granular computing (GrCS) to enable the problem of crowd segmentation to be conceptualized at different levels of granularity, and to map problems into computationally tractable subproblems. It shows that by exploiting the correlation among pixel granules, we are able to aggregate structurally similar pixels into meaningful atomic structure granules. This is useful in outlining natural boundaries between crowd and background (i.e., noncrowd) regions. From the structure granules, we infer the crowd and background regions by granular information classification. GrCS is scene-independent and can be applied effectively to crowd scenes with a variety of physical layout and crowdedness. Extensive experiments have been conducted on hundreds of real and synthetic crowd scenes. The results demonstrate that by exploiting the correlation among granules, we can outline the natural boundaries of structurally similar crowd and background regions necessary for crowd segmentation.

In this paper, a new formula of fuzzy Caputo fractional-order derivatives (0 <; v ≤ 1) in terms of shifted Chebyshev polynomials is derived. The proposed approach introduces a shifted Chebyshev operational matrix in combination with a shifted Chebyshev tau technique for the numerical solution of linear fuzzy fractional-order differential equations. The main advantage of the proposed approach is that it simplifies the problem alike in solving a system of fuzzy algebraic linear equations. An approximated error bound between the exact solution and the proposed fuzzy solution with respect to the number of fuzzy rules and solution errors is derived. Furthermore, we also discuss the convergence of the proposed method from the fuzzy perspective. Experimentally, we show the strength of the proposed method in solving a variety of fractional differential equation models under uncertainty encountered in engineering and physical phenomena (i.e., viscoelasticity, oscillations, and resistor-capacitor (RC) circuits). Comparisons are also made with solutions obtained by the Laguerre polynomials and the fractional Euler method.

A fuzzy rough set theory based gait decision system is presented for diagnosis of fall risk. Distracted walking is investigated. Gait cycles of thirty young participants were monitored while they walked across a walkway at a self-selected pace and also while being distracted by performing a task alongside walking. Results shows distracted walking being similar to impaired walking of an elderly person. The aim is to adopt instance selection methods to eliminate redundant or noisy sample from training data thus preventing the system from detecting erroneous gait patterns or deviations. Decision systems have great potential in medical informatics to serve as diagnostic tools. Results show that instance selection improves the efficacy of various classifiers in detecting distracted gait. The model is capable of assessing gait based on the easily obtainable features and categorizing them in terms of fall risk. This will help to identify individuals at risk in fall prevention management.

In this paper, we propose a novel approach for the numerical solution of fuzzy fractional differential equations (FFDEs) under fuzzy Caputo-type derivative. More specifically, we first obtain the equivalent integral form of original problem, then the fractional integral equation is approximated using Laplace transforms. Afterwards, we can get the solution by employing any numerical method. Indeed, the proposed approach introduces an efficient and practical way to solve a wide range of fractional models under uncertainty. The most important advantage of this procedure is that the complexity of dealing with the fractional derivative is removed from the calculations, which can reduce the computational costs, considerably. Illustrative examples address the validity and appropriateness of this technique.

“Ceci n'est pas une pipe” French for “This is not a pipe”. This is the description painted on the first painting in the figure above. But to most of us, how could this painting is not a pipe, at least not to the great Belgian surrealist artist Rene Magritte. He said that the painting is not a pipe, but rather an image of a pipe. In this paper, we present a study on large-scale classification of fine-art paintings using the Deep Convolutional Network. Our objectives are two-folds. On one hand, we would like to train an end-to-end deep convolution model to investigate the capability of the deep model in fine-art painting classification problem. On the other hand, we argue that classification of fine-art collections is a more challenging problem in comparison to objects or face recognition. This is because some of the artworks are non-representational nor figurative, and might requires imagination to recognize them. Hence, a question arose is that does a machine have or able to capture “imagination” in paintings? One way to find out is train a deep model and then visualize the low-level to high-level features learnt. In the experiment, we employed the recently publicly available large-scale “Wikiart paintings” dataset that consists of more than 80,000 paintings and our solution achieved state-of-the-art results (68%) in overall performance.

Blur is a common artifact in video, which adds more complexity to text detection and recognition. To achieve good accuracies for text detection and recognition, this paper suggests a new method for classifying blurred and non-blurred frames in video. We explore quality metrics, namely, BRISQUE, NRIQA, GPC and SI, in a new way for classification. We estimate the values of these metrics with the help of predefined samples called reference values. To widen the difference between metric values for better classification, we introduce scaling factors as a non-linear sigmoidal function, which considers the metric of each current frame and its reference and results in templates. Based on the characteristics of metrics, the proposed method finds a relationship between the metrics to derive rules for classification. To classify the frame containing local blur, we explore quad tree division with classification rules which divide non-blurred blocks to identify local blur. We use standard databases, namely, ICDAR 2013, ICDAR 2015 and YVT videos for experimentation, and evaluate the proposed method in terms of text detection and recognition rates given by text detection and binarization methods before and after classification.

The main study of this paper is focused on the solutions of a class of fuzzy sequential fractional differential equations in the form of (0Dxβy)'(x) = b(x)y(x), where (0Dxβy)(x) is the fuzzy Riemann-Liouville derivative of order β ∈ (0, 1). On this subject, a new fuzzy complete metric space is introduced. Finally, we proof the existence and uniqueness of our solution using the contraction principle.

Humans have the natural capabilities to perceive and anticipate actions of objects they interact with, including incidents happen within their neighborhood. These days, this important aspect of human perception has been widely incorporated in the computer vision framework to perform human action detection task. However, little attention is paid to the problem of detecting ongoing human actions as early as possible, which is crucial in a number of important applications ranging from video surveillance to health-care. In this paper, we propose a framework for detecting ongoing human actions as early as possible, i.e. detecting an action as soon as it begins, but before it completes. This is make possible with the used of Fuzzy Bandler and Kohout's sub-triangle product (BK subproduct) inference mechanism, utilizing the fuzzy capabilities in handling the arisen uncertainties during the human action recognition stage for a reliable decision making. Experimental results on publicly available dataset illustrate the effectiveness of the proposed method.

This paper studies convolutional neural networks (CNN) to learn unsupervised feature representations for 44 different plant species, collected at the Royal Botanic Gardens, Kew, England. To gain intuition on the chosen features from the CNN model (opposed to a ‘black box’ solution), a visualisation technique based on the deconvolutional networks (DN) is utilized. It is found that venations of different order have been chosen to uniquely represent each of the plant species. Experimental results using these CNN features with different classifiers show consistency and superiority compared to the state-of-the art solutions which rely on hand-crafted features.

Object recognition systems usually require fully complete manually labeled training data to train classifier. In this paper, we study the problem of object recognition, where the training samples are missing during the classifier learning stage, a task also known as zero-shot learning. We propose a novel zero-shot learning strategy that utilizes the topic model and hierarchical class concept. Our proposed method advanced where cumbersome human annotation stage (i.e., attribute-based classification) is eliminated. We achieve comparable performance with state-of-the-art algorithms in four public datasets: PubFig (67.09%), Cifar-100 (54.85%), Caltech-256 (52.14%), and Animals with Attributes (49.65%), when unseen classes exist in the classification task.

Fractional calculus has a long successful history of 300 years, as it able to model natural phenomena states more accurately than the differential equations of integer order. With this, it plays an important role in variant disciplines. Recently, variant fractional models for the Bloch equations have been proposed, however, effective numerical methods for the fractional Bloch equation (FBE) are still in the infancy stage. In this paper, we extend the time-fractional Bloch equation (TFBE) to fuzzy field under the generalized Caputo differentiability, such that these extensions have natural relationship between crisp. For this purpose, we adopted the fractional Adams-Bashforth-Moulton (FABM) type predictorcorrector method, and introduced a new variant - the fuzzy fractional ADM (FFABM) to find the numerical solution. In this case, a new theorem concerning the error of our proposed FFADM method is also presented. Finally, the capability of the newly developed numerical methods is demonstrated in a fuzzy fractional-order problem, and it achieves satisfactorily in terms of numerical stability.

Image understanding is an important research domain in the computer vision due to its wide real-world applications. For an image understanding framework that uses the Bag-of-Words model representation, the visual codebook is an essential part. Random forest (RF) as a tree-structure discriminative codebook has been a popular choice. However, the performance of the RF can be degraded if the local patch labels are poorly assigned. In this paper, we tackle this problem by a novel way to update the RF codebook learning for a more discriminative codebook with the introduction of the soft class labels, estimated from the pLSA model based on a feedback scheme. The feedback scheme is performed on both the image and patch levels respectively, which is in contrast to the state-of-the-art RF codebook learning that focused on either image or patch level only. Experiments on 15-Scene and C-Pascal datasets had shown the effectiveness of the proposed method in image understanding task.

It is common for CCTV operators to overlook interesting events taking place within the crowd due to large number of people in the crowded scene (i.e. marathon, rally). Thus, there is a dire need to automate the detection of salient crowd regions acquiring immediate attention for a more effective and proactive surveillance. This paper proposes a novel framework to identify and localize salient regions in a crowd scene, by transforming low-level features extracted from crowd motion field into a global similarity structure. The global similarity structure representation allows the discovery of the intrinsic manifold of the motion dynamics, which could not be captured by the low-level representation. Ranking is then performed on the global similarity structure to identify a set of extrem a. The proposed approach is unsupervised so learning stage is eliminated. Experimental results on public datasets demonstrates the effectiveness of exploiting such extrem a in identifying salient regions in various crowd scenarios that exhibit crowding, local irregular motion, and unique motion areas such as sources and sinks.

Ambiguity or uncertainty is a pervasive element of many real-world decision-making processes. Variation in decisions is a norm in this situation when the same problem is posed to different subjects. Psychological and metaphysical research has proven that decision making by humans is subjective. It is influenced by many factors such as experience, age, background, etc. Scene understanding is one of the computer vision problems that fall into this category. Conventional methods relax this problem by assuming that scene images are mutually exclusive; therefore, they focus on developing different approaches to perform the binary classification tasks. In this paper, we show that scene images are nonmutually exclusive and propose the fuzzy qualitative rank classifier (FQRC) to tackle the aforementioned problems. The proposed FQRC provides a ranking interpretation instead of binary decision. Evaluations in terms of qualitative and quantitative measurements using large numbers and challenging public scene datasets have shown the effectiveness of our proposed method in modeling the nonmutually exclusive scene images.

In an attempt to incorporate user knowledge in order to decide about the equality of sets, the concepts of approximate equalities using rough sets were introduced. These notions have been generalised in several ways and very recently [1] extended four types of approximate equalities using rough fuzzy sets instead of only rough sets. To be precise, a concept of leveled approximate equality was introduced and properties were studied. In this paper we extend this work with case studies to illustrate the applications of the concepts and compare them respectively. We also introduce and discuss the rough measures of basic sets, fuzzy sets and interpret four types of approximate equalities in terms of the accuracy measure as well as rough measures. The analysis had provided a clear distinguish notion in terms of the measures.

Recently, Bandler-Kohout (BK) subproduct based reasoning scheme has been a popular choice in various kind of applications. In this paper, we aim to enhance the BK subproduct based reasoning schemes in two aspects: (1) Extend the BK subproduct in term of Interval Type-2 Fuzzy Sets (IT2FS) instead of the ordinary Type-1 Fuzzy Sets (T1FS), and (2) Introduce weight parameter to the reasoning scheme. Firstly, studies have shown that IT2FS have better capability in handling data with uncertainty compare to the ordinary T1FS. Thus, we extend the BK subproduct in terms of IT2FS theory where subsethood measure based on the fuzzy implication operators for the IT2FS has been developed. Secondly, weight parameter associated to each features is introduced to form a weighted inference scheme with the BK subproduct. The introduction of the weight parameter is to aid in distinguishing the influence of different features in the reasoning process. In here, the Linguistic Weighted Average (LWA) is adopted to solve the outputs of this weighted reasoning scheme. Finally, a case study is employed to demonstrate the capability of the proposed approach.

To be able to perform human action recognition from multiple views is a great challenge in the field of computer vision. State-of-the-art solutions have been focusing on building a 3D action model from multiple views in a multi, calibrated cameras' environment. Promising results were achieved; however, these approaches tend to assume that human action is performed frontal-parallel to each of the multiple cameras. In a real world scenario, this is not always true and the overlapping regions in such systems are very limited. In this paper, we proposed a fuzzy action recognition framework for multiple views within a single camera. We adopted fuzzy quantity space in the framework and introduced a new concept called the Signature Action Behaviour to model an action from multiple views and represent it as fuzzy descriptor. Then, distance measure is applied to deduce an action. Experimental results showed the efficiency of our proposed framework in modeling the actions from different viewpoints and styles.

Current zero-shot learning methods relied on attributes to describe the unseen class characteristics, using the learned seen class model. However, these approaches required extensive attribute labels on each object class, and a well-defined, attributes relationship between the seen and unseen class with the aid of human knowledge. In this work, we avoid these with a novel learning process using the probabilistic Latent Semantic Analysis (pLSA). We replace the attributes with topic model and extend the representation as a mapping algorithm to object classes, so that zero-shot learning would be possible. With this, less annotated class information is required to achieve similar performance. Evaluations on three public datasets had shown the effectiveness of our proposed method.

Scene classification has been studied extensively in the recent past. Most of the state-of-the-art solutions assumed that scene classes are mutually exclusive. However, this is not true as a scene image may belongs to multiple classes and different people are tend to respond inconsistently even given a same scene image. In this paper, we propose a fuzzy qualitative approach to address this problem. That is, we first adopted the fuzzy quantity space to model the training data. Secondly, we present a novel weight function, w to train a fuzzy qualitative scene model in the fuzzy qualitative states. Finally, we introduce fuzzy qualitative partition to perform the scene classification. Empirical results using a standard dataset and a comparison with K-nearest neighbour has shown the effectiveness and robustness of the proposed method.

Multi descriptions property is a fundamental property on membership functions of fuzzy sets, which has not been attended so far. The property suggests that not all membership functions of fuzzy sets can be viewed as a simple number in the range of 0 and 1. In fact, membership function of an element in a fuzzy set should be viewed as the sum of strength of the element that show each attribute which describes the fuzzy set. Therefore, fuzzy sets operation such as union, intersection and subsethood measure need to be revised. By adopting multi descriptions property, this paper introduced a set of improvement based on Checklist Paradigm for these operations for type-1 fuzzy sets. Since the multi descriptions property may also appear in type-2 fuzzy sets, we proposed a method for the subsethood measurement, namely Representative Method. This Representative Method provides simple and fast approximate measurements for subsethood.

Depth information is a very important cue to understand human motion. In this paper, we establish that, even with no real depth camera, the concept of obtaining depth information is applicable for human motion detection. We propose a new motion detection method based on the concept of a real world video surveillance system enhanced with depth cameras. It is developed for detecting and analysing human motion. First, it imitates depth measuring process of a depth camera. Specially chosen in the image during the initialization process, view points play the role of cameras, whereas the depth is measured as a distance from these points to the human figure in the image. Initially, the body is partitioned into four segments to obtain the information about which part of the body is moving. Then, in course of the working cycle of the method, the received depth values are constantly subtracted from the previously obtained values, and the intensity of the body motion is calculated using root mean square. The method has been tested on actions taken from a standard motion dataset (IXMAS). It proved to be stable and reliable.

A reliable human skin detection method that is adaptable to different human skin colors and illumination conditions is essential for better human skin segmentation. Even though different human skin-color detection solutions have been successfully applied, they are prone to false skin detection and are not able to cope with the variety of human skin colors across different ethnic. Moreover, existing methods require high computational cost. In this paper, we propose a novel human skin detection approach that combines a smoothed 2-D histogram and Gaussian model, for automatic human skin detection in color image(s). In our approach, an eye detector is used to refine the skin model for a specific person. The proposed approach reduces computational costs as no training is required, and it improves the accuracy of skin detection despite wide variation in ethnicity and illumination. To the best of our knowledge, this is the first method to employ fusion strategy for this purpose. Qualitative and quantitative results on three standard public datasets and a comparison with state-of-the-art methods have shown the effectiveness and robustness of the proposed approach.

A reliable human skin detection method that is adaptable to different human skin colors and illumination conditions is essential for better human skin segmentation. Even though different human skin-color detection solutions have been successfully applied, they are prone to false skin detection and are not able to cope with the variety of human skin colors across different ethnic. Moreover, existing methods require high computational cost. In this paper, we propose a novel human skin detection approach that combines a smoothed 2-D histogram and Gaussian model, for automatic human skin detection in color image(s). In our approach, an eye detector is used to refine the skin model for a specific person. The proposed approach reduces computational costs as no training is required, and it improves the accuracy of skin detection despite wide variation in ethnicity and illumination. To the best of our knowledge, this is the first method to employ fusion strategy for this purpose. Qualitative and quantitative results on three standard public datasets and a comparison with state-of-the-art methods have shown the effectiveness and robustness of the proposed approach.

Recognition of human posture is one step in the process of analyzing human behaviour. However, it is an ill-defined problem due to the high degree of freedom exhibited by the human body. In this paper, we study both supervised and unsupervised learning algorithms to recognise human posture in image sequences. In particular, we are interested in a specific set of postures which are representative of typical applications found in video analytics. The algorithms chosen for this paper are Kmeans, artificial neural network, self organizing maps and particle swarm optimization. Experimental results have shown that the supervised learning algorithms outperform the unsupervised learning algorithms in terms of the number of correctly classified postures. Our future work will focus on detecting abnormal behaviour based on these recognised static postures.

Understanding actions is a complex issue in many aspects. However, most of the literature on action recognition deals with only simple actions. In this paper, we proposed the fuzzy qualitative robot kinematics framework to complex actions over time, e.g. walk then run and over the body, walk while wave hand etc. The human limbs is modelled as articulated rigid bodies and its motion is represented by a series of such models in terms of time. With this, we eventually converted the human motion analysis into a conventional robotic problem which has been well studied. Experimental results has shown that the action model built in this manifold offers few advantages. e.g. handles the tradeoffs in the off-the-shelf tracking algorithm and avoid using generative model where the size of the training data typically goes as the square of the number of states.

The understanding of human behaviour in video is a challenging task in that the same behaviour might have several different meanings depending upon the scene and task context in which it is performed. While human seem to perform scene interpretations without effort, this is a formidable and yet unsolved task for artificial vision systems. One of the main reasons is that there exists a gap between low-level vision at signal level and high-level representation of activities at symbolic level. In this paper, we present an intelligent connection framework using Gaussian mixture model-based clustering (GMM) to bridge the low-level vision data and the qualitative normalised templates (QNT) - a symbolic representation for human motion based on fuzzy qualitative robot kinematics, which could link the former with domain-dependent scenarios. The proposed method has been applied to the recognition of eight types of human motions and an empirical comparison with fuzzy hidden Markov-based human motion recognition system.

This paper proposes a fuzzy qualitative approach to vision-based human motion analysis with an emphasis on human motion recognition. It achieves feasible computational cost for human motion recognition by combining fuzzy qualitative robot kinematics with human motion tracking and recognition algorithms. First, a data-quantization process is proposed to relax the computational complexity suffered from visual tracking algorithms. Second, a novel human motion representation, i.e., qualitative normalized template, is developed in terms of the fuzzy qualitative robot kinematics framework to effectively represent human motion. The human skeleton is modeled as a complex kinematic chain, and its motion is represented by a series of such models in terms of time. Finally, experiment results are provided to demonstrate the effectiveness of the proposed method. An empirical comparison with conventional hidden Markov model (HMM) and fuzzy HMM (FHMM) shows that the proposed approach consistently outperforms both HMMs in human motion recognition.

The understanding of human motions captured in image sequences pose two main difficulties which are often regarded as computationally ill-defined: 1) modelling the uncertainty in the training data, and 2) constructing a generic activity representation that can describe simple actions as well as complicated tasks that are performed by different humans. In this paper, these problems are addressed from a direction which utilises the concept of fuzzy qualitative robot kinematics [9]. First of all, the training data representing a typical activity is acquired by tracking the human anatomical landmarks in an image sequences. Then, the uncertainty arise when the limitations of the tracking algorithm are handled by transforming the continuous training data into a set of discrete symbolic representations - qualitative states in a quantisation process. Finally, in order to construct a template that is regarded as a combination ordered sequence of all body segments movements, robot kinematics, a well-defined solution to describe the resulting motion of rigid bodies that form the robot, has been employed. We defined these activity templates as qualitative normalised templates, a manifold trajectory of unique state transition patterns in the quantity space. Experimental results and a comparison with the hidden Markov models have demonstrated that the proposed method is very encouraging and shown a better successful recognition rate on the two available motion databases.

Classification of human motion in video data is essential in numerous applications. However, problems arise as the human exhibits complex and dynamic motion that is nonlinear and time varying. In this paper, we propose a knowledge-based human motion classification framework that employs fuzzy qualitative reasoning to address these problems. Our approach utilises the rich contextual information (e.g. structural and transitional characteristic of human motion) captured in video sequence to effectively study and recognise human motion. With the aid of domain knowledge, a set of fuzzy rules are defined in the knowledge base. This work is in contrast with previous attempts that depend solely on the trajectories of the body parts. Experimental results on two classes of motion (e.g. walking and running) that result in similar motions; and a comparison with the conventional method has demonstrated and validated the effectiveness of the proposed method in improving the perception of human motion.