The scarcity of reliable, extensive datasets hampers the training of effective models for wearable healthcare technology. This data gap frequently introduces biases into training sets, which then carry over into the models themselves. Such inherent biases pose substantial fairness challenges, particularly in sensitive healthcare scenarios. To this end, we propose AFairDNet, an effective active learning framework that utilizes a small collection of annotated data to create an initial classifier, and then continually refines it by incorporating synthesized ‘hard’ signals, representing areas where the model's training is currently insufficient. To ensure both creativity and ethical responsibility in these generated signals, we enhance the signal generation process using Chain of Thought (CoT) reasoning. The model employs real-time iterative CoT refinement of the model's text prompts to condition the multisensor signal diffuser, ensuring that the synthesized multisensor biosignals are not only of high quality but also semantically faithful. Extensive evaluations using two large publicly available multisensor emotion recognition datasets demonstrate that by leveraging a small yet comprehensive collection of synthesized samples (i.e., around 1.4% of the total training set), AFairDNet may boost a baseline classifier's performance, outperforming the state-of-the-art methods. More precisely, in addition to achieving$1.5-3 \%$higher accuracy than current supervised and self-supervised baselines, AFairDNet also boasts an impressive Total Fairness Score, signaling its potential for more responsible and transparent AI-driven synthesized signal generation.

Gaining insight into the brain's visual representation through reconstructing what we see from brain activity is of immense importance and interest. Though fMRI and MEG achieve high-quality image reconstruction and classification, their cost and size restrict broader real-world applications, particularly outside clinical settings. In contrast, although Electroencephalography (EEG) is a cost-effective, non-invasive tool producing high temporal resolution signals, it remains less explored primarily due to its susceptibility to noise and complex spatio-temporal characteristics. To address these, we propose BrainDiffNet, an effective EEG-to-Image generation model that leverages a subject's contextual and EEG spatio-temporal information to guide a fine-tuned Stable Diffusion model, resulting in highquality, semantically relevant images from brain activity. A robust Temporal Masked Autoencoder, designed for high-resolution EEG, enables the model to effectively extract features and manage noisy or incomplete EEG query representations. Indepth evaluation using the large-scale EEG-ImageNet dataset demonstrates the outperformance of BrainDiffNet in both tasks: Object Classification and Image Reconstruction. In fact, the model significantly outperforms state-of-the-art baseline methods, achieving a$15-20 \%$higher accuracy in classification across all granularity levels and a$7-12 \%$improvement in all featurespecific two-way identification metrics for image reconstruction.

Emotion recognition’s efficacy for individual and group conversations hinges upon integrating data from diverse modalities (e.g., text, audio, and video). However, the intrinsic heterogeneity of these data sources, coupled with the interpersonal contextual influences in groups, present significant challenges to accurately identifying emotion. Furthermore, during inference most applications cannot guarantee the presence of all modalities the model was trained on. Towards this, we propose Missing Information robust Joint Transformer (MIJoT), which is designed to capture the extensive cross-modal spatio-temporal context in a dialogue. Unlike a majority of existing works, which rely on a fundamental assumption that a full set of modalities is present during inference, MIJoT’s Masked Mode-specific Data Modelling ensures consistent performance in presence of partial modality details. By enabling powerful masked modeling pretraining, MIJoT delivers an impressive ability to deal with a wide range of incomplete modality scenarios. We also utilize a Contrastive Imbalance Loss that maximizes inter-class distance on a high-dimensional hyper-sphere, to improve performance on minority classes. An effective multimodal fusion enables the model to analyze both utterances and dialogue (sequence of utterances) between multiple people within an integrated computation framework and deliver superior performance (4-12%) improvement in weighted-F1 scores) compared to the current best-performing methods across several large-scale public datasets. Additionally, MIJoT with incomplete modalities report comparable performance to existing state-of-the-art techniques which use complete multi-mode information, and thereby demonstrate its robustness to partial mode information of the queries.

Evaluating speaker emotion in conversations is crucial for various applications requiring human-computer interaction. However, co-occurrences of multiple emotional states (e.g. 'anger' and 'frustration' may occur together or one may influence the occurrence of the other) and their dynamic evolution may vary dramatically due to the speaker's internal (e.g., influence of their personalized socio-cultural-educational and demographic backgrounds) and external contexts. Thus far, the previous focus has been on evaluating only the dominant emotion observed in a speaker at a given time, which is susceptible to producing misleading classification decisions for difficult multi-labels during testing. In this work, we present Self-supervised Multi- Label Peer Collaborative Distillation (SeMuL-PCD) Learning via an efficient Multimodal Transformer Network, in which complementary feedback from multiple mode-specific peer networks (e.g.transcript, audio, visual) are distilled into a single mode-ensembled fusion network for estimating multiple emotions simultaneously. The proposed Multimodal Distillation Loss calibrates the fusion network by minimizing the Kullback-Leibler divergence with the peer networks. Additionally, each peer network is conditioned using a self-supervised contrastive objective to improve the generalization across diverse socio-demographic speaker backgrounds. By enabling peer collaborative learning that allows each network to independently learn their mode-specific discriminative patterns,SeMUL-PCD is effective across different conversation environments. In particular, the model not only outperforms the current state-of-the-art models on several large-scale public datasets (e.g., MOSEI, EmoReact and ElderReact), but with around 17% improved weighted F1-score in the cross-dataset experimental settings. The model also demonstrates an impressive generalization ability across age and demography-diverse populations.

Rapid dissemination of misinformation is a major societal problem receiving increasing attention. Unlike Deep-fake, Out-of-Context misinformation, in which the unaltered unimode contents (e.g. image, text) of a multi-modal news sample are combined in an out-of-context manner to generate deception, requires limited technical expertise to create. Therefore, it is more prevalent a means to confuse readers. Most existing approaches extract features from its uni-mode counterparts to concatenate and train a model for the misinformation classification task. In this paper, we design a self-supervised feature representation learning strategy that aims to attain the multi-task objectives: (1) task-agnostic, which evaluates the intra- and inter-mode representational consistencies for improved alignments across related models; (2) task-specific, which estimates the category-specific multi-modal knowledge to enable the classifier to derive more discriminative predictive distributions. To compensate for the dearth of annotated data representing varied types of misinformation, the proposed Self-Supervised Distilled Learner (SSDL) utilizes a Teacher network to weakly guide a Student network to mimic a similar decision pattern as the teacher. The two-phased learning of SSDL can be summarized as: initial pretraining of the Student model using a combination of contrastive self-supervised task-agnostic objective and supervised task-specific adjustment in parallel; finetuning the Student model via self-supervised knowledge distillation blended with the supervised objective of decision alignment. In addition to the consistent out-performances over the existing baselines that demonstrate the feasibility of our approach, the explainability capacity of the proposed SSDL also helps users visualize the reasoning behind a specific prediction made by the model.

Recommendations form an integral part of instructional process and can be instrumental to promote or maintain student engagement in various course-offering platforms. Automated evaluation of a student’s discussion forum post and proactively generating a personalized recommendation to address the student’s learning requirement are of huge interest, specifically in an in-person classroom setting, which is still considered to be a dominant mode of mainstream learning. However, the task is growingly challenging due to the ever-expanding enrollment trend, where students from the wider socio-economic backgrounds become more of the norm. The traditional support structures, such as daytime-only office hours for student advisement, are typically inadequate. Toward this, we propose a multi-modal attentive learning framework that keeps track of the temporally evolving student learning patterns and their conversation dynamics in the course discussion board to automatically estimate relevant expression (e.g. ‘confusion’, ‘question’, ‘urgency’) reflected in forum posts. Based on the classifier evaluation, the consequential content recommendation module employs information propagation on a multi-view course specific knowledge graph to obtain a more context-aware entity embedding for recommendation. The system derives a personalized ranked list or relevant documents/video clippings augmented with the explainability score that enables the system to reveal the recommendation justifications for an improved student acceptance. The experimental results, which leverage our in-house course-specific multi-modal activity details from three large in-person Undergraduate and Postgraduate level STEM courses, demonstrate the effectiveness of our approach.

Human emotion recognition plays a pivotal role in building an intelligent conversational agent for providing real-time automated support service in various problem settings. Recent research works have explored the temporal patterns in conversations to enable a comprehensive understanding of the content and context of conversations from a video clip, which does not fully leverage the multi-modal (facial expressions of the participants, speech tone, content, and context of the discussion) information and their temporal evolution. To address this, we propose a multimodal attentive learning framework that keeps track of spatio-temporal states of the participants and their conversation dynamics. By designing a novel contrastive loss-based optimization framework, the proposed method exhibits promise in identifying the emotion state of the individual speaker in real-time and can identify top-k words in the conversation that influence emotion recognition. The consistent superior performance over other state-of-the-art works in two large-scale datasets, MELD and IEMOCAP, demonstrate the feasibility of our approach.

Automatic detection of malicious or provocative social media content is increasingly important to efforts leading to counteract illicit activities in the digital platform. This paper proposes a context-aware, regional feature representation learning framework that exploits multimodal cues to improve automatic detection by enhancing classification via accurate localization. Unlike most existing multimodal approaches, which evaluate category membership of multimedia web contents only at the image level, our approach leverages object proposals to identify potential interest regions within images in order to provide more precise localization performance that in turn, in a context-aware setting, also improves classification result. The proposed attention learning module estimates the domain specific cross-modal, fine-grained regional feature correspondence, conditioned on the classification categories, by evaluating mode relevance scores in a data-driven manner. The initial classification decision is further validated using a query adaptive decision fine-tuning for a more accurate final prediction. Experiments on publicly available datasets and on in-house datasets demonstrate superior classification performance as compared to monomodal and existing multimodal baselines.

In this paper, we propose an effective, multi-view, generative, transfer learning framework for multivariate time-series data. While generative models are demonstrated effective for several machine learning tasks, their application to time-series classification problems is underexplored. The need for additional exploration is motivated when data are large, annotations are unbalanced or scarce, or data are distributed and fragmented. Recent advances in computer vision attempt to use synthesized samples with system generated annotations to overcome the lack or imbalance of annotated data. However, in multi-view problem settings, view mismatches between the synthetic data and real data pose additional challenges against harnessing new annotated data collections. The proposed method offers important contributions to facilitate knowledge sharing, while simultaneously ensuring an effective solution for domain-specific, finelevel categorizations. We propose a principled way to perform view adaptation in a cross-view learning environment, wherein pairwise view similarity is identified by a smaller subset of source samples that closely resemble the target data patterns. This approach integrates generative models within a deep classification framework to minimize the gap between source and target data. More precisely, we design category specific conditional, generative models to update the source generator in order for transforming source features so that they appear as target features and simultaneously tune the associated discriminative model to distinguish these features. During each learning iteration, the source generator is conditioned by a source training set represented as some target-like features. This transformation in appearance was performed via a target generator specifically learned for target-specific customization per category. Afterward, a smaller source training set, indicating close target pattern resemblance in terms of the corresponding generative and discriminative loss, is used to fine-tune the source classification model parameters. Experiments show that compared to existing approaches, our proposed multiview, generative, transfer learning framework improves timeseries classification performance by around 4% in the UCI multiview activity recognition dataset, while also showing a robust, generalized representation capacity in classifying several large-scale multi-view light curve collections.

In this paper, we propose an effective, multi-view, deep, transfer learning framework for multivariate time-series data. Though widely used for tasks such as computer vision, the application of transfer learning to time-series classification problems (e.g., classification of light curves) is underexplored. The proposed framework makes several important contributions to facilitate knowledge sharing, while simultaneously ensuring an effective solution for domain specific fine-level categorizations. First, in contrast to the traditional approaches, the proposed framework describes pairwise view similarity by identifying a smaller subset of source-view samples that closely resemble the target data patterns. Second, by means of two-phase learning, a generic baseline model is learned on a larger source data collection and later fine-tuned on a smaller target data collection, precisely approximating the target data patterns. Third, an effective view-adaptive timestamp weighting scheme evaluates the relative importance of each timestamp in a more data-driven manner, which enables a more flexible yet discriminative feature representation scheme in the presence of evolving data characteristics. As shown by experiments, compared to the existing approaches, our proposed deep transfer learning framework improves classification performance by around 2-3% in the UCI multi-view activity recognition dataset, while also showing a robust, generalized representation capacity in classifying several large-scale multi-view light curve collections.

In this paper, we propose an effective, multi-view, multivariate deep classification model for time-series data. Multi-view methods show promise in their ability to learn correlation and exclusivity properties across different independent information resources. However, most current multi-view integration schemes employ only a linear model and, therefore, do not extensively utilize the relationships observed across different view-specific representations. Moreover, the majority of these methods rely exclusively on sophisticated, handcrafted features to capture local data patterns and, thus, depend heavily on large collections of labeled data. The multi-view, multivariate deep classification model for time-series data proposed in this paper makes important contributions to address these limitations. The proposed model derives a LSTM-based, deep feature descriptor to model both the view-specific data characteristics and cross-view interaction in an integrated deep architecture while driving the learning phase in a data-driven manner. The proposed model employs a compact context descriptor to exploit view-specific affinity information to design a more insightful context representation. Finally, the model uses a multi-view factor-pooling scheme for a context-driven attention learning strategy to weigh the most relevant feature dimensions while eliminating noise from the resulting fused descriptor. As shown by experiments, compared to the existing multi-view methods, the proposed multi-view deep sequential learning approach improves classification performance by roughly 4% in the UCI multi-view activity recognition dataset, while also showing significantly robust generalized representation capacity against its single-view counterparts, in classifying several large-scale multi-view light curve collections.

Sketch-based object search is a challenging problem mainly due to three difficulties: 1) how to match the primary sketch query with the colorful image; 2) how to locate the small object in a big image that is similar to the sketch query; and 3) given the large image database, how to ensure an efficient search scheme that is reasonably scalable. To address the above challenges, we propose leveraging object proposals for object search and localization. However, instead of purely relying on sketch features, we propose fully utilizing the appearance features of object proposals to resolve the ambiguities between the matching sketch query and object proposals. Our proposed query adaptive search is formulated as a subgraph selection problem, which can be solved by the maximum flow algorithm. By performing query expansion, it can accurately locate the small target objects in a cluttered background or densely drawn deformation-intensive cartoon (Manga like) images. To improve the computing efficiency of matching proposal candidates, the proposed Multi View Spatially Constrained Proposal Selection encodes each identified object proposal in terms of a small local basis of anchor objects. The results on benchmark datasets validate the advantages of utilizing both the sketch and appearance features for sketch-based search, while ensuring sufficient scalability at the same time.

Recently, global features aggregated from local convolutional features of the convolutional neural network have shown to be much more effective in comparison with hand-crafted features for image retrieval. However, the global feature might not effectively capture the relevance between the query object and reference images in the object instance search task, especially when the query object is relatively small and there exist multiple types of objects in reference images. Moreover, the object instance search requires to localize the object in the reference image, which may not be achieved through global representations. In this paper, we propose a Fuzzy Objects Matching (FOM) framework to effectively and efficiently capture the relevance between the query object and reference images in the dataset. In the proposed FOM scheme, object proposals are utilized to detect the potential regions of the query object in reference images. To achieve high search efficiency, we factorize the feature matrix of all the object proposals from one reference image into the product of a set of fuzzy objects and sparse codes. In addition, we refine the feature of the generated fuzzy objects according to its neighborhood in the feature space to generate more robust representation. The experimental results demonstrate that the proposed FOM framework significantly outperforms the state-of-the-art methods in precision with less memory and computational cost on three public datasets.

Cyber threat intelligence and security informatics play critical roles in the identification of the principal influencers and threats associated with criminal activity and extremism in web-based communities. This paper presents an effective, dynamic learning framework utilizing domain specific context information to design a novel classification model that can robustly identify malicious social media posts (e.g. enrollment propaganda for extremist groups) with expressions of extremism or criminal intent. Research towards automated identification of extreme online posts and their associated key suspects and threats faces numerous challenges, 1) Online data, particularly social media data, originated from numerous independent and heterogeneous sources are largely unstructured. 2) The tactics, techniques, and procedures (TTPs) of criminal activity and extremism are constantly evolving. 3) There are limited ground truth data to support the development of effective classification technologies. In this paper, we present a human-machine collaborative, semi-supervised learning system that can efficiently and effectively identify malicious social media posts in presence of these challenges. Our system and framework develops an initial classifier from limited annotated data and in an interactive manner evolves dynamically into a sophisticated model using shortlisted relevant samples, identified via a graph-based optimization method, solvable by maximum flow algorithm. This same method also may be used to refine the classifier as TTPs evolve. Under this framework, the classifier performance converges faster using roughly 1-2 orders of magnitude fewer annotated samples, as compared to fully supervised solutions, resulting in a reasonably acceptable accuracy of nearly 80%. We validate our framework using a large collection of English and non-English flagged words extracted from three web-based forums and manually verified by multiple independent annotators.

The objective of a news veracity detection system is to identify various types of potentially misleading or false information, typically in a digital platform. A critical challenge in this scenario is that there are large volumes of data available online. However, obtaining samples with annotations (i.e. ground-truth labels) is difficult and a known limiting factor for many data analytic tasks including the current problem of news veracity detection. In this paper, we propose a human-machine collaborative learning system to evaluate the veracity of a news content, with a limited amount of annotated data samples. In a semi-supervised scenario, an initial classifier is learnt on a small, limited amount of the annotated data followed by an interactive approach to gradually update the model by shortlisting only relevant samples from the large pool of unlabeled data that are most likely to improve the classifier performance. Our prioritized active learning solution achieves faster convergence in terms of the classification performance, while requiring about 1–2 orders of magnitude fewer annotated samples compared to fully supervised solutions to attain a reasonably acceptable accuracy of nearly 80%. Unlike traditional deep learning architecture, the proposed active learning based deep model designed with a smaller number of more localized filters per layer can efficiently learn from small relevant sample batches that can effectively improve performance in the weakly-supervised learning environment and thus is more suitable for several practical applications. An effective dynamic domain adaptive feature weighting scheme can adjust the relative importance of feature dimensions iteratively. Insightful initial feedback gathered from two independent learning modules (a NLP shallow feature based classifier and a deep classifier), modeled to capture complementary information about data characteristics are finally fused together to achieve an impressive 25% average gain in the detection performance.

Big data analytics traditionally involves download of massive amounts of datasets to common server/cluster for processing. Analytic process gets slower with increasing size of required data and network conditions. Data scientists also need explicit access to data locations to download required data. Explicit access to required data may not always be granted due to security reasons. To simplify and accelerate the analytics process on distributed big data with security considerations, we proposed the Virtual Information Fabric Infrastructure (VIFI) for data driven discoveries. Instead of moving large amounts of data to a common place of processing, VIFI allows automatic transfer of required analytics programs to the distributed data locations for in-place processing of relevant data. VIFI allows data scientists to conduct and coordinate complex analytics processes on distributed data repositories using containerization technology and open-source workflow design tools. VIFI alleviates users from having detailed knowledge of distributed data locations, as well as required dependencies, installation and configuration of analytical libraries. In this paper, we demonstrate our current and future work to improve the VIFI architecture using previous and additional uses cases, data management layer that simplifies search of relevant data sets through addition of metadata, integration with security policies at different institutions with the proposed VIFI security layer, and the use of a user-friendly web interface to carry different VIFI activities.

Most existing binary embedding methods prefer compact binary codes (b-dimensional) to avoid high computational and memory cost of projecting high-dimensional visual features (d-dimensional, b

This paper proposes a context-aware, graph-based approach for identifying anomalous user activities via user profile analysis, which obtains a group of users maximally similar among themselves as well as to the query during test time. The main challenges for the anomaly detection task are: (1) rare occurrences of anomalies making it difficult for exhaustive identification with reasonable false-alarm rate, and (2) continuously evolving new context-dependent anomaly types making it difficult to synthesize the activities apriori. Our proposed query-adaptive graph-based optimization approach, solvable using maximum flow algorithm, is designed to fully utilize both mutual similarities among the user models and their respective similarities with the query to shortlist the user profiles for a more reliable aggregated detection. Each user activity is represented using inputs from several multi-modal resources, which helps to localize anomalies from time-dependent data efficiently. Experiments on public datasets of insider threats and gesture recognition show impressive results.

Data analytics is being increasingly used in cyber-security problems, and found to be useful in cases where data volumes and heterogeneity make it cumbersome for manual assessment by security experts. In practical cyber-security scenarios involving data-driven analytics, obtaining data with annotations (i.e. ground-truth labels) is a challenging and known limiting factor for many supervised security analytics task. Significant portions of the large datasets typically remain unlabelled, as the task of annotation is extensively manual and requires a huge amount of expert intervention. In this paper, we propose an effective active learning approach that can efficiently address this limitation in a practical cyber-security problem of Phishing categorization, whereby we use a human-machine collaborative approach to design a semi-supervised solution. An initial classifier is learnt on a small amount of the annotated data which in an iterative manner, is then gradually updated by shortlisting only relevant samples from the large pool of unlabelled data that are most likely to influence the classifier performance fast. Prioritized Active Learning shows a significant promise to achieve faster convergence in terms of the classification performance in a batch learning framework, and thus requiring even lesser effort for human annotation. An useful feature weight update technique combined with active learning shows promising classification performance for categorizing Phishing/malicious URLs without requiring a large amount of annotated training samples to be available during training. In experiments with several collections of PhishMonger's Targeted Brand dataset, the proposed method shows significant improvement over the baseline by as much as 12%.

Sketch-based object search is a challenging problem mainly due to two difficulties: (1) how to match the binary sketch query with the colorful image, and (2) how to locate the small object in a big image with the sketch query. To address the above challenges, we propose to leverage object proposals for object search and localization. However, instead of purely relying on sketch features, e.g., Sketch-a-Net, to locate the candidate object proposals, we propose to fully utilize the appearance information to resolve the ambiguities among object proposals and refine the search results. Our proposed query adaptive search is formulated as a sub-graph selection problem, which can be solved by maximum flow algorithm. By performing query expansion using a smaller set of more salient matches as the query representatives, it can accurately locate the small target objects in cluttered background or densely drawn deformation intensive cartoon (Manga like) images. Our query adaptive sketch based object search on benchmark datasets exhibits superior performance when compared with existing methods, which validates the advantages of utilizing both the shape and appearance features for sketch-based search.

While there has been a significant amount of work on object search and image retrieval, the focus has primarily been on establishing effective models for the whole images, scenes, and objects occupying a large portion of an image. In this paper, we propose to leverage object proposals to identify small and smooth-structured objects in a large image database. Unlike popular methods exploring a coarse image-level pairwise similarity, the search is designed to exploit the similarity measures at the proposal level. An effective graph-based query expansion strategy is designed to assess each of these better matched proposals against all its neighbors within the same image for a precise localization. Combined with a shape-aware feature descriptor EdgeBoW, a set of more insightful edge-weights and node-utility measures, the proposed search strategy can handle varying view angles, illumination conditions, deformation, and occlusion efficiently. Experiments performed on a number of other benchmark datasets show the powerful and superior generalization ability of this single integrated framework in dealing with both clutter-intensive real-life images and poor-quality binary document images at equal dexterity.

We propose a graph-based optimization framework to leverage category independent object proposals (candidate object regions) for logo search in a large scale image database. The proposed contour-based feature descriptor EdgeBoW is robust to view-angle changes, varying illumination conditions and can implicitly capture the significant object shape information. Having been equipped with a local descriptor, it can handle a fair amount of occlusion and deformation frequently present in a real-life scenario. Given a small set of initially retrieved candidate object proposals, a fast graph-based short-listing scheme is designed to exploit the mutual similarities among these proposals for eliminating outliers. In contrast to a coarse image-level pairwise similarity measure, this search focussed on a few specific image regions provides a more accurate method for matching. The proposed query expansion strategy aims to assess each of the remaining better matched proposals against all its neighbors within the same image for a precise localization. Combined with an efficient feature descriptor EdgeBoW, a set of more insightful edge-weights and node-utility measures can yield promising results, specially for object categories primarily defined by its shape. Extensive set of experiments performed on a number of benchmark datasets demonstrates its effectiveness and superior generalization ability in both clutter intensive real-life images and poor quality binary document images.