Image editing using generative models has recently advanced through GAN- and diffusion-based techniques. While the current image manipulation methods shows considerable performance in general image editing, their effectiveness drops when extended to fashion attribute editing. This is due to multiple challenges such as category-specific and overlapping attributes, inherent entanglement in real-world datasets that leads to degraded editing quality and unintended attribute shifts. To address these challenges, we propose FAE-Net (Fashion Attribute Editing Network), a latent diffusion framework that leverages disentangled latent projections for precise and reliable attribute manipulation. Our method first disentangles the latent projections to mitigate the inherent entanglement in the data and then conditions the diffusion model with those projections to improve the manipulation control in the presence of overlapping attributes. The attribute presence detector in FAE-Net handles category-specific attributes and prevents invalid attribute manipulations during inference. Extensive experiments on three large-scale datasets demonstrate that our proposed method achieves more controllable, disentangled, and faithful attribute editing compared to state-of-the-art methods.

The growing use of wearable devices requires accurate and compact representations of high dimensional physiological signals. This work presents a UNet inspired autoencoder to represent and reconstruct multiple neuro-physiological signals from single channel data. The architecture comprises single-encoder/dual-branched decoders to obtain self-attention enabled compact embeddings of mixed ExG (EEG /ECG) signals through decaying encoder-decoder skip connections, for improved representation capability. The embeddings are separable into individual ExG components enabling simultaneous reconstruction of high fidelity EEG and ECG sources. The pretrained encoder can be used for a complex downstream task with minimum fine-tuning. Using the proposed method on a large corpus of single-channel mixed ExG generated from overnight Polysomnography (PSG) recordings, we show subject- and class- independent EEG/ECG reconstructions validated by multiple domain-specific metrics, and evaluate the classification performance of the encoded EEG embeddings into five sleep stages as a downstream task.

Automatic fashion attribute recognition enables retailers to address an array of applications. Usually, fashion attributes are manually input in the system by retailers, which is a time-consuming and an error prone process. To alleviate this, several existing works use traditional CNN-based backbones to recognize attributes. These backbones generate attribute embeddings that are entangled in the feature space. Existing methods that generate disentangled attribute embedding do not explicitly specify the location of attributes, and often extract features from irrelevant regions. This directly impacts the quality of downstream tasks. To alleviate this problem, we have proposed a novel framework to extract location-aware attribute representation using localization maps created from fashion landmarks. These localization maps highlight regions of interest in an image, aiding localized attribute feature extraction. Moreover, we have proposed a novel fusion module to effectively select important features from the global representation of an image to enhance the local features of the attribute. These attribute embeddings are then used in downstream applications such as attribute recognition, hierarchical taxonomy classification, and retrieval with two large-scale datasets. Using the proposed model, we observe improvement in performance from the state-of-the-art by a significant margin.

Scientific objectives from earth observation to astronomy require high-resolution observations from space-based platforms. However, designing space telescopes with large primary apertures to achieve high-resolution and high Signal-to-Noise Ratio observations, especially for those operating in longer wavelengths (like Thermal Infrared, TIR), is not feasible due to difficulties in manufacturing, launching, and post-deployment stabilizing. This work proposes three novel lightweight, optically-sparse (also known as partially-filled) mirrors with non-uniform sub-aperture sizes. These designs reduce the mass of the primary mirror and its supporting framework. The crux of these designs is, however, significant suppression of sidelobes in the resulting Point Spread Functions (PSFs). The study includes restored images and image quality indices, demonstrating the effectiveness of such lightweight unequal sub-apertures as replacements for large monolithic mirrors with only a marginal loss in performance.

Automatic object recoloring or swatch generation aims to change color of an object or the entire scene without altering the structural consistency. The challenges exacerbate while dealing with retail items, such as clothing, shoes and accessories due to the presence of complex patterns, folds and shadows, deformation caused by the human model, and small components such as frills, buttons, belts, etc. The challenge associated with this application increases further for multi-colored item and multi-apparel setup. In this work, we aim to address these problems with our novel architecture SwatchNet based on generative adversarial network (GAN). It stems from the proposed apparel components aware feature extraction module to create rich feature embedding, which guides the proposed dual attention u-net to synthesize recolored product image. For seamless information flow, we have also proposed a dual attention module at the bottleneck of encoder and decoder. Finally, we unify a diverse set of recoloring applications using a fixed training and inference pipeline. The experimental results of fashion item recoloring for several test setups using four large-scale datasets demonstrate the effectiveness of our proposed approach.

Recommendation systems improve users' online shopping experience by recommending relevant items from a large pool of items in different categories. Fashion recommendation systems apart from recommending individual fashion items also recommend fashion outfits. In this work, we consider the problem of the outfit compatibility prediction task, an integral part of the fashion outfit recommendation system. A compatibility prediction module determines whether all the items in an outfit are visually compatible with each other and match the user's preferences. Existing approaches can be grouped based on the representation scheme: (i) pair-wise and (ii) set or sequence. Pair-wise representation does not consider the outfit as a whole, and the sequence representation approaches are sensitive to the ordering of the items. Further, these methods do not explicitly capture the visual relationship between the items. We propose a novel method for the personalized outfit-compatible prediction task. The proposed method represents the outfit as a graph and uses a dot-attention graph neural network to capture the visual relationship between items. The graph read-out layer generates the final outfit embedding. A novel approach is proposed to model the user's preference for different styles. The final outfit compatibility score is generated by computing the similarity between outfit embedding and user embedding. Experimental results and ablation study on the Polyvore-U dataset, highlight the effectiveness of the proposed method.

Analysis of Electroencephalography (EEG) signals for everyday in-situ applications is hindered by many challenges including artifacts induced from physiological and environmental sources as well as anatomical factors. Recent studies have proposed analytical frameworks for the assessment of scalp EEG quality and identification of artifacts using rule based methods. These methods are typically used in an offline processing manner, employed before signal analysis when abundant data is available. With the advent of wearable devices, it is important to build techniques that work on short term signals giving us the ability of intervention based on the signal quality. Further, ear-EEG is a new modality that requires assessment and calibration with short term signals. To support this new modality we conducted a detailed study of scalp and ear-EEG data from the perspective of different measures as well as time duration. An algorithm is developed to identify the epochs with EOG and EMG artifacts in the ear-EEG using a set of metrics based on the characteristics of EEG. Thresholds of these metrics are determined by training on a dataset containing synchronous ear and scalp EEG with good results. The algorithm obtained an accuracy of 76.7% and 76.84% in classifying artifact EEG for scalp referenced ear-EEG and re-referenced ear-EEG, respectively.

Tracking of inventory and rearrangement of mis-placed items are some of the most labor-intensive tasks in a retail environment. While there have been attempts at using vision-based techniques for these tasks, they mostly use planogram compliance for detection of any anomalies, a technique that has been found lacking in robustness and scalability. Moreover, existing systems rely on human intervention to perform corrective actions after detection. In this paper, we present Co-AD, a Concept-based Anomaly Detection approach using a Vision Transformer (ViT) that is able to flag misplaced objects without using a prior knowledge base such as a planogram. It uses an auto-encoder architecture followed by outlier detection in the latent space. Co-AD has a peak success rate of 89.90% on anomaly detection image sets of retail objects drawn from the RP2K dataset, compared to 80.81% on the best-performing baseline of a standard ViT auto-encoder. To demonstrate its utility, we describe a robotic mobile manipulation pipeline to autonomously correct the anomalies flagged by Co-AD. This work is ultimately aimed towards developing autonomous mobile robot solutions that reduce the need for human intervention in retail store management.

Fashion landmark detection is a fundamental task in several fashion image analysis problems.The associated challenges involving non-rigid structures and variations in style and orientation makes it extremely hard to accurately detect the landmarks.In this paper, we propose Appearance-Context network (ACNet), which encapsulates both global and local contextual information extending the axial attention mechanism.We design axial attention augmented local appearance network and introduce a novel Global-Context aware axial attention module which aggregates the global features attending discriminatory cues across height, width and channel axes.The proposed ACNet architecture outperforms existing methods on two large-scale fashion landmark datasets.

Fusing SAR and multi-spectral images to generate a precise land cover map in a weakly supervised setting is a challenging yet essential problem. The inaccurate, noisy, and inexact ground truth labels pose difficulty training any machine learning models. In this paper, we make a fundamental and pivotal contribution towards improving the ground truth label quality using domain knowledge. We present a simple yet effective mechanism to refine the low-resolution noisy ground truth labels. The proposed approach is trained and tested on a publicly available DFC2020 dataset. Through experiments, we show the effectiveness of our method by training a deep learning model on the refined labels that outperform even the models trained with clean ground truth.

In this paper, we present a novel few-shot cross-sensor domain adaptation technique between SAR and multispectral data for LULC classification. Cross-sensor, such as SAR and multispectral, domain adaptation is a long standing challenge in remote sensing. Due to scarcity of large annotated dataset for every domain, it is desirable to have a method that enables cross-domain training with limited supervisory signal in that domain. We address this problem in this paper with a novel few-shot domain adaptation technique. We leverage large corpus of annotated multispectral dataset to improve performance for SAR based LULC classification. We propose a novel Feature Domain Alignment (FDA) loss function to align higher dimension features between multispectral and SAR domain. We validate our approach in publicly available DFC2020 dataset and achieve 78% overall LULC classification accuracy using only 5% annotated SAR samples.

Scene graph captures rich semantic information of an image by representing objects and their relationships as nodes and edges of a graph. Recent works have demonstrated that scene graph representation improves the performance of various computer vision tasks such as image retrieval, action recognition, visual question answering. Computationally efficient scene graph generation methods are required to leverage scene graphs in various real-world applications (e.g., autonomous driving, robotics). A typical scene graph generation model consists of two modules: (i) object detector and (ii) scene graph classifier. The scene graph classifier module predicts the object category and object-object relationships. The presence of a quadratic number of potential edges poses a major challenge in the scene graph classification task. Detecting the relationship between each object pair using the traditional approach is computationally intensive and non-scalable. To address this issue, we propose a novel module named EdgeNet that directly predicts the set of relevant edges and helps to prune out a significant number of unrelated object pairs, thereby improving the effectiveness and efficiency of the scene graph classifier. The proposed EdgeNet is a generic module and can be plugged into an existing scene graph classifier. Experimental results highlight the effectiveness and efficiency of the proposed approach on the Visual Genome dataset.

Large-scale attribute recognition in fashion retail images is a crucial task in image-based recommendation systems. The challenges are due to the visually-similar instances, localized minute information and overlapping features. Moreover, the class imbalance further exacerbates the challenge, needing for a specific solution to alleviate the problem. In this work, F-AttNet architecture is proposed, which is designed by the hierarchical alignment of the novel Attentive Multi-scale Feature (AMF) encoder blocks. AMF encoders extract mid-level multi-scale fine-grained attribute features involving multiple representations of low-level features and finally, the high-level global description is encoded by adaptively calibrating the channel weights. For improving the training performance, a novel gamma-variant focal loss is developed to handle class imbalance by assigning more penalty and assigning relative weights to positive and negative instances shifting the focus of the network to false instances. Experimental results and ablation studies of F-AttNet using a large-scale fashion attribute recognition database iMaterialist-2018 demonstrate significant performance improvement than the state-of-the-art methodologies.

In this paper, we present a deep convolutional neural network (CNN) architecture for segmenting semantic changes between two images. The main objective is to segment changes at the semantic level than detecting background changes, which are irrelevant to the application. The difficulties include seasonal changes, lighting differences, artifacts due to alignment and occlusion. The existing approaches fail to address all the problems together; thus, none of them achieve state-of-the-art performance in three publicly available change detection datasets: VL-CMU-CD [1], TSUNAMI [2] and GSV [2]. Our proposed approach is a simple yet effective method that can handle even adverse challenges. In our approach, we leverage the correlation between high-level abstract CNN features to segment the changes. Compared with several traditional and other deep learning-based change detection methods, our proposed method achieves state-of-the-art performance in all three datasets.

Drones are used in a number of industrial applications such as asset tracking and inspection. Indoor industrial applications based on visual data pose various challenges such as low lighting conditions and presence of non-planar scenes. Due to the nature of the indoor applications, image data is acquired at close range and this leads to the loss of context. In order to get global context, image stitching is a key step for data interpretation. We propose an approach to stitch drone-captured indoor video frames, where feature based stitching fails. In order to achieve this, the image feature data extracted is fused with drone inertial measurement unit (IMU) data. The approach is tested in a warehouse and the performance is compared with other state-of-the-art image stitching algorithms. The proposed approach shows robust performance in cases of highly non-planar scenes.

Dictionary learning (DL) is one of the popular sparse coding machine learning techniques. In image processing literature, every input image is represented as the sparse linear combination of basis vectors. DL has been shown to have wide applications for image restoration as well as pattern recognition problems. In DL, the input image is factorized into dictionary and sparse codes. This factorization always leaves a residual or approximation error. Very few works in the literature had focused on to leverage the information present in this residual. In this paper, we use residuals within our framework and show that the restoration performance or accurate prediction of missing label in multi-spectral images can be significantly improved over conventional DL based techniques. We initially show that the higher order frequencies are propagated through residuals. Then we show that incorporating this residual in the image restoration methodology can significantly improve the outcomes. Finally, we propose a technique to solve the problem of missing label prediction by using a restoration based deep residual dictionary learning framework.

Real-time environment monitoring and analysis is an important research area of Internet of Things (IoT). Understanding the behavior of the complex ecosystem requires analysis of detailed observations of an environment over a range of different conditions. One such example in urban areas includes the study of tree canopy cover over the microclimate environment using heterogeneous sensor data. There are several challenges that need to be addressed, such as obtaining reliable and detailed observations over monitoring area, detecting unusual events from data, and visualizing events in real-time in a way that is easily understandable by the end users (e.g., city councils). In this regard, we propose an integrated geovisualization framework, built for real-time wireless sensor network data on the synergy of computational intelligence and visual methods, to analyze complex patterns of urban microclimate. A Bayesian maximum entropy-based method and a hyperellipsoidal model-based algorithm have been build in our integrated framework to address above challenges. The proposed integrated framework was verified using the dataset from an indoor and two outdoor network of IoT devices deployed at two strategically selected locations in Melbourne, Australia. The data from these deployments are used for evaluation and demonstration of these components' functionality along with the designed interactive visualization components.

Infrastructure detection and monitoring is a difficult task. Due to the advances in unmanned vehicles and image analytics, it is possible to decrease the human effort and achieve consistent results in infrastructure assessments using aerial image processing. Reliable detection and integrity checking of power infrastructure including conductor lines, pylons and insulators in a diverse background is the most challenging task in drone based automatic infrastructure monitoring. Most techniques in literature use first principle approach that tries to represent the image as features of interest. This paper proposes a deep learning approach for power infrastructure detection. Graph based post processing is applied for improving the outcomes of the generated deep model. A f-score of 75% is achieved using the deep model which is further improved using spectral clustering for the conductor lines, pylons and insulators that form the core parts of power infrastructure.

Vision is one of the most advanced and important sensory input in humans. However, many people have vision problems due to birth defects, uncorrected errors, work nature, accidents, and aging. The white cane and guide dog are the most widely used means of navigation for the vision-impaired. With advancements in technology, electronic devices have been created using different sensors and technologies to help navigate the blind. Electronic Travel Aids (ETAs) assist in navigating a person by collecting information about the environment and relaying this information in a form that allows a blind or vision-impaired person to understand the nature of the environment. However, there is still a lack of devices to detect potholes and uneven pavements, which inhibits mobility after dark. This pilot study proposes a computer vision based pothole and uneven surface detection approach to assist blind people in meeting their mobility needs. The system includes projecting laser patterns, recording the patterns through a monocular video, analyzing the patterns to extract features and then providing path cues for the blind user. With over 90% accuracy in detecting potholes, the proposed system aims to assist blind people in real-time navigation.

Analyzing crowd events in a video is key to understanding the behavioral characteristics of people (humans). Detecting crowd events in videos is challenging because of articulated human movements and occlusions. The aim of this paper is to detect the events in a probabilistic framework for automatically interpreting the visual crowd behavior. In this paper, crowd event detection and classification in optical flow manifolds (OFMs) are addressed. A new algorithm to detect walking and running events has been proposed, which uses optical flow vector lengths in OFMs. Furthermore, a new algorithm to detect merging and splitting events has been proposed, which uses Riemannian connections in the optical flow bundle (OFB). The longest vector from the OFB provides a key feature for distinguishing walking and running events. Using a Riemannian connection, the optical flow vectors are parallel transported to localize the crowd groups. The geodesic lengths among the groups provide a criterion for merging and splitting events. Dispersion and evacuation events are jointly modeled from the walking/running and merging/splitting events. Our results show that the proposed approach delivers a comparable model to detect crowd events. Using the performance evaluation of tracking and surveillance 2009 dataset, the proposed method is shown to produce the best results in merging, splitting, and dispersion events, and comparable results in walking, running, and evacuation events when compared with other methods.

Epilepsy is one of the most common neurological disorders and patients suffer from unprovoked seizures. In contrast, psychogenic nonepileptic seizures (PNES) are another class of seizures that are involuntary events not caused by abnormal electrical discharges but are a manifestation of psychological distress. The similarity of these two types of seizures poses diagnostic challenges that often leads in delayed diagnosis of PNES. Further, the diagnosis of PNES involves high-cost hospital admission and monitoring using video-electroencephalogram machines. A wearable device that can monitor the patient in natural setting is a desired solution for diagnosis of convulsive PNES. A wearable device with an accelerometer sensor is proposed as a new solution in the detection and diagnosis of PNES. The seizure detection algorithm and PNES classification algorithm are developed. The developed algorithms are tested on data collected from convulsive epileptic patients. A very high seizure detection rate is achieved with 100% sensitivity and few false alarms. A leave-one-out error of 6.67% is achieved in PNES classification, demonstrating the usefulness of wearable device in the diagnosis of PNES.

The growth in pervasive network infrastructure called the Internet of Things (IoT) enables a wide range of physical objects and environments to be monitored in fine spatial and temporal detail. The detailed, dynamic data that are collected in large quantities from sensor devices provide the basis for a variety of applications. Automatic interpretation of these evolving large data is required for timely detection of interesting events. This article develops and exemplifies two new relatives of the visual assessment of tendency (VAT) and improved visual assessment of tendency (iVAT) models, which uses cluster heat maps to visualize structure in static datasets. One new model is initialized with a static VAT/iVAT image, and then incrementally (hence inc-VAT/inc-iVAT) updates the current minimal spanning tree (MST) used by VAT with an efficient edge insertion scheme. Similarly, dec-VAT/dec-iVAT efficiently removes a node from the current VAT MST. A sequence of inc-iVAT/dec-iVAT images can be used for (visual) anomaly detection in evolving data streams and for sliding window based cluster assessment for time series data. The method is illustrated with four real datasets (three of them being smart city IoT data). The evaluation demonstrates the algorithms’ ability to successfully isolate anomalies and visualize changing cluster structure in the streaming data.

Increasing population density in urban centers demands adequate provision of services and infrastructure to meet the needs of city inhabitants, encompassing residents, workers, and visitors. The utilization of information and communications technologies to achieve this objective presents an opportunity for the development of smart cities, where city management and citizens are given access to a wealth of real-time information about the urban environment upon which to base decisions, actions, and future planning. This paper presents a framework for the realization of smart cities through the Internet of Things (IoT). The framework encompasses the complete urban information system, from the sensory level and networking support structure through to data management and Cloud-based integration of respective systems and services, and forms a transformational part of the existing cyber-physical system. This IoT vision for a smart city is applied to a noise mapping case study to illustrate a new method for existing operations that can be adapted for the enhancement and delivery of important city services.

A lip-reading technique that identifies visemes from visual data only and without evaluating the corresponding acoustic signals is presented. The technique is based on vertical components of the optical flow (OF) analysis and these are classified using support vector machines (SVM). The OF is decomposed into multiple non-overlapping fixed scale blocks and statistical features of each block are computed for successive video frames of an utterance. This technique performs automatic temporal segmentation (i.e., determining the start and the end of an utterance) of the utterances, achieved by pair-wise pixel comparison method, which evaluates the differences in intensity of corresponding pixels in two successive frames. The experiments were conducted on a database of 14 visemes taken from seven subjects and the accuracy tested using five and ten fold cross validation for binary and multiclass SVM respectively to determine the impact of subject variations. Unlike other systems in the literature, the results indicate that the proposed method is more robust to inter-subject variations with high sensitivity and specificity for 12 out of 14 visemes. Potential applications of such a system include human computer interface (HCI) for mobility-impaired users, lip reading mobile phones, in-vehicle systems, and improvement of speech based computer control in noisy environment.

Obstructive sleep apnea or hypopnea causes a pause or reduction in airflow with continuous breathing effort. The aim of this study is to identify individual apnea and hypopnea events from normal breathing events using wavelet-based features of 5-s ECG signals (sampling rate = 250 Hz) and estimate the surrogate apnea index (AI)/hypopnea index (HI) (AHI). Total 82,535 ECG epochs (each of 5-s duration) from normal breathing during sleep, 1638 ECG epochs from 689 hypopnea events, and 3151 ECG epochs from 1862 apnea events were collected from 17 patients in the training set. Two-staged feedforward neural network model was trained using features from ECG signals with leave-one-patient-out cross-validation technique. At the first stage of classification, events (apnea and hypopnea) were classified from normal breathing events, and at the second stage, hypopneas were identified from apnea. Independent test was performed on 16 subjects' ECGs containing 483 hypopnea and 1352 apnea events. The cross-validation and independent test accuracies of apnea and hypopnea detection were found to be 94.84% and 76.82%, respectively, for training set, and 94.72% and 79.77%, respectively, for test set. The Bland-Altman plots showed unbiased estimations with standard deviations of +/- 2.19, +/- 2.16, and +/- 3.64 events/h for AI, HI, and AHI, respectively. Results indicate the possibility of recognizing apnea/hypopnea events based on shorter segments of ECG signals.

Knowledge of the secondary structure and solvent accessibility of a protein plays a vital role in prediction of fold, and eventually the tertiary structure of the protein. This paper deals with prediction of relative solvent accessibility, given only the amino-acid sequence. In this paper, we use an improved support vector regression (SVR) and new kernels for real valued prediction of solvent accessibility. In this regard, two main issues are addressed. First we address the problem of e selection, which we found to be somewhat problematic in our earlier work (e is a parameter with significant influence on noise insensitivity and generalization of SVRs). In particular, rather than employ the standard trial and error based approach, we used an improved tube shrinking method to find e. Secondly, a novel kernel combining solvation model, electrostatic charge model and evolutionary information in the form of position specific scoring matrix (PSSM) is given. A new dataset of 472 proteins with less than 20% sequence identity is curated and used to evaluate the result. To make a more objective comparison with earlier methods, we use a standard dataset and show that the proposed scheme is better than the ones normally used in literature. We also report a lowest mean absolute error (MAE) so far of 0.12 on the standard dataset.

Knowledge of the secondary structure and solvent accessibility of a protein plays a vital role in the prediction of fold, and eventually the tertiary structure of the protein. A challenging issue of predicting protein secondary structure from sequence alone is addressed. Support vector machines (SVM) are employed for the classification and the SVM outputs are converted to posterior probabilities for multi-class classification. The effect of using Chou–Fasman parameters and physico-chemical parameters along with evolutionary information in the form of position specific scoring matrix (PSSM) is analyzed. These proposed methods are tested on the RS126 and CB513 datasets. A new dataset is curated (PSS504) using recent release of CATH. On the CB513 dataset, sevenfold cross-validation accuracy of 77.9% was obtained using the proposed encoding method. A new method of calculating the reliability index based on the number of votes and the Support Vector Machine decision value is also proposed. A blind test on the EVA dataset gives an average Q3accuracy of 74.5% and ranks in top five protein structure prediction methods. Supplementary material including datasets are available on .