The emergence of bias in deep neural models represents a significant reliability concern, which may lead to overoptimistic results on seen data while compromising the model's ability to generalize effectively on unseen datasets. Recent studies conducted on The Cancer Genome Atlas (TCGA), which is a publicly available repository of histopathology images, reveal that the TCGA cancerous features extracted by deep neural networks surprisingly are able to discriminate slides based on their origin sites. This finding undoubtedly indicates the existence of site-specific patterns embedded in the extracted features learned by deep networks rather than focusing on histomorphologic patterns. Consequently, this biased behavior raises concerns about the reliability of these networks. This observation motivates us to conduct a series of experiments in which we present two distinct evolutionary feature selection strategies, each differentiated by its objective function evaluation. The primary goal is to select the features with a minimized foot-print of data source signatures, thereby ensuring a more accurate and site-independent cancer classification. We have conducted nine comprehensive independent experiments across nine cancer types, employing each evolutionary strategy. The comparison between results obtained through evolutionary strategies and the original feature sets highlights the substantial impact of feature selection methods on bias reduction while maintaining accuracy in cancer-type discrimination. Furthermore, the comparison of the two strategies with each other demonstrates the intricate nature of bias and its integration with cancerous features during the training process.

Feature selection plays a pivotal role in the data preprocessing and model-building pipeline, significantly en-hancing model performance, interpretability, and resource efficiency across diverse domains. In population-based optimization methods, the generation of diverse individuals holds utmost importance for adequately exploring the problem landscape, particularly in highly multi-modal multi-objective optimization problems. Our study reveals that, in line with findings from sev-eral prior research papers, commonly employed crossover and mutation operations lack the capability to generate high-quality diverse individuals and tend to become confined to limited areas around various local optima. This paper introduces an augmen-tation to the diversity of the population in the well-established multi-objective scheme of the genetic algorithm, NSGA-II. This enhancement is achieved through two key components: the genuine initialization method and the substitution of the worst individuals with new randomly generated individuals as a re-initialization approach in each generation. The proposed multi-objective feature selection method undergoes testing on twelve real-world classification problems, with the number of features ranging from 2,400 to nearly 50,000. The results demonstrate that replacing the last front of the population with an equivalent number of new random individuals generated using the genuine initialization method and featuring a limited number of features substantially improves the population's quality and, consequently, enhances the performance of the multi-objective algorithm.

The increasing accessibility of legal corpora and databases create opportunities to develop data-driven techniques and advanced tools that can facilitate a variety of tasks in the legal domain, such as legal search and research, legal document review and summary, legal contract drafting, and legal outcome prediction. Compared with other application domains, the legal domain is characterized by the huge scale of natural language text data, the high complexity of specialist knowledge, and the critical importance of ethical considerations. The MLLD workshop aims to bring together researchers and practitioners to share the latest research findings and innovative approaches in employing data mining, machine learning, information retrieval, and knowledge management techniques to transform the legal sector. Building upon the previous successes, the third edition of the MLLD workshop will emphasize the exploration of new research opportunities brought about by recent rapid advances in Large Language Models and Generative AI. We encourage submissions that intersect computer science and law, from both academia and industry, embodying the interdisciplinary spirit of CIKM.

This “blue sky idea” paper outlines the opportunities and challenges in data mining and machine learning involving making a computational attorney — an intelligent software agent capable of helping human lawyers with a wide range of complex high-level legal tasks such as drafting legal briefs for the prosecution or defense in court. In particular, we discuss what a ChatGPT-like Large Legal Language Model (L3M) can and cannot do today, which will inspire researchers with promising short-term and long-term research objectives.

This full-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and epistemic uncertainty for text classification models. Then, we describe several state-of-the-art approaches to uncertainty quantification and analyze their scalability to big text data: Virtual Ensemble in GBDT, Bayesian Deep Learning (including Deep Ensemble, Monte-Carlo Dropout, Bayes by Backprop, and their generalization Epistemic Neural Networks), Evidential Deep Learning (including Prior Networks and Posterior Networks), as well as Distance Awareness (including Spectral-normalized Neural Gaussian Process and Deep Deterministic Uncertainty). Next, we talk about the latest advances in uncertainty quantification for pre-trained language models (including asking language models to express their uncertainty, interpreting uncertainties of text classifiers built on large-scale language models, uncertainty estimation in text generation, calibration of language models, and calibration for in-context learning). After that, we discuss typical application scenarios of uncertainty quantification in text classification (including in-domain calibration, cross-domain robustness, and novel class detection). Finally, we list popular performance metrics for the evaluation of uncertainty quantification effectiveness in text classification. Practical hands-on examples/exercises are provided to the attendees for them to experiment with different uncertainty quantification methods on a few real-world text classification datasets such as CLINC150.

Optimization problems widely arise in various science and engineering fields and can be computationally expensive in many real-world applications. Evaluation of the fitness function to assess a candidate solution is the main operation in all optimization procedures which can be heavily compute-intensive. Machine learning-based surrogate models can contribute to learning the specific pattern among the decision variables and objective values to consequently reduce the computation time of fitness evaluation. In this study, we have proposed a novel pairwise surrogate model to identify the superiority between candidate solutions in a pairwise comparison despite the fact that most of the surrogate models try to predict the exact fitness value. The proposed idea can significantly help the optimizer to reach better results in a shorter period of time. It seems comparing two candidate solutions for a greedy selection is much easier than approximating fitness values for both. We demonstrated Graph Neural Network (GNN) for this purpose to be trained on a limited number of pairwise ranks and then utilized to compare a pair of candidate solutions. In order to examine the efficacy of our model, we utilized different well-known single-objective optimization benchmarks in dimensions 10,20, and 30. Moreover, the results of the learning-based evaluation are compared with the results from the real fitness evaluation. The results, assessed in terms of the number of fitness calls and the best-found solution, showed that the proposed method is able to decrease the computing cost of fitness evaluation significantly while we achieve a comparable solution. Our model can be tested with any optimization algorithm which employs a comparison-based mechanism among its candidate solutions.

Generally speaking, labeled data is difficult and expensive to provide for applications in machine learning and data mining. One of the earliest approaches to tackle this problem is semi-supervised self-training to take advantages of labeled and unlabeled data to create pseudo-labeled data. However, reaching a high level of confidence to predict the unseen data by a classifier with a limited number of labeled samples is a challenging task. Generating pseudo-labeled data can collaboratively improve the self-training to increase its confidence for further predictions. This paper proposes a collaborative framework between augmentation and self-training to accurately train a model with very limited labeled data. Our framework includes two components where the first component is an unsupervised technique to augment labeled data and feed to the second component which is self-training. The first component uses a custom variational autoencoder architecture (VAE) to generate new samples by adding randomly generated noise to encoded latent representation. As a result, the augmentation component can generate the unique and unexplored images with respect to limited input data distribution. We evaluated the proposed framework on handwritten MNIST image dataset. The conducted experiment shows that the generative component can be helpful in overcoming the problem of inaccurate self-training prediction when sufficient labeled data is not accessible.

Multi-objective optimization is a branch of computation to solve mathematical optimization problems when conflicting multiple objective functions must be simultaneously optimized. Many population-based algorithms, such as NSGA-II are used to find an optimal Pareto set of solutions in a short time. However, previous research works showed the performance of classic NSGA-II is degraded when solving problems with many objectives (M ≥ 5). In this work, we aim to take advantage of center-based sampling scheme to increase the exploration and exploitation capability of NSGA-II algorithm. This sampling strategy has demonstrated promising results on several single-objective evolutionary algorithms such as GA, DE, and PSO. Recently, a novel clustering center-based strategy has been proposed, which motivated us to utilize center-based sampling scheme in NSGA-II to solve multi-objective optimization problems. The outcomes confirm that the proposed clustering center-based NSGA-II is able to effectively solve CEC-2017 multi-objective benchmark problems with 2, 3, 5, 10, and 15 objectives.

A parsimonious convolutional neural network (CNN) for text document classification that replicates the ease of use and high classification performance of linear methods is presented. This new CNN architecture can leverage locally trained latent semantic analysis (LSA) word vectors. The architecture is based on parallel 1D convolutional layers with small window sizes, ranging from 1 to 5 words. To test the efficacy of the new CNN architecture, three balanced text datasets that are known to perform exceedingly well with linear classifiers were evaluated. Also, three additional imbalanced datasets were evaluated to gauge the robustness of the LSA vectors and small window sizes. The new CNN architecture consisting of 1 to 4-grams, coupled with LSA word vectors, exceeded the accuracy of all linear classifiers on balanced datasets with an average improvement of 0.73%. In four out of the total six datasets, the LSA word vectors provided a maximum classification performance on par with or better than word2vec vectors in CNNs. Furthermore, in four out of the six datasets, the new CNN architecture provided the highest classification performance. Thus, the new CNN architecture and LSA word vectors could be used as a baseline method for text classification tasks.

Winning a game in professional sports is the most significant matter for a team. All teams strive to bring their best performance to a game, and this requires considering all the possible lineups which coaches have available. Therefore, determining the lineup is more and more significant for a team in their winning endeavour. The ongoing result during a game defines the next decision coaches have to make to maintain or improve the outcome. Adaptive changes in a lineup of a team requires a complex decision making system. This system must consider the advantages, drawbacks, and previous experience about both teams' performance under similar situations. In order to analyze and predict lineups' performance, the authors create a directed, weighted, and signed network of all lineups that teams use against each other from 2007-2016 seasons in National Basketball Association (NBA) games. The proposed model uses machine learning and network analysis techniques to predict the performance of a lineup under a given situation by utilizing graph theory and Inverse Squared Metric.

In this paper, learning social networks from incomplete relationship data is proposed. Link prediction is addressed as a semi-supervised learning problem where the task is to predict a larger part of networks using available knowledge of smaller parts. By this assumption, social network extraction is translated into a classification problem. While in real case scenarios majority of links are unknown, we hypothesis self-training as the most common semi-supervised learning method can provide an effective approach for learning from unlabelled data. We proposed an interpolative self-training technique that leverages node information to generate a set of examples in learning phase along with their connections as their associated labels. The approach generates data by interpolation of documents assigned to a pair of nodes. Documents as the implicit content shared between the individual nodes provide a scope for the estimation of their similarities. Then generated training data are employed in a link prediction model with two different scenarios. The first scenario interprets the link prediction as a conventional classification problem in which we have examples from both positive (link) and negative (no-link) classes. However, the second scenario addresses more realistic case where only some positive examples (links or connections) are known. Social networks are usually very sparse structures. The sparsity of social networks implies that in the classification framework of link prediction, we deal with an imbalance class distribution in which among all possible links there are a few connections (positive class) vs. many disconnections (negative class). In order to deal with class skew and enhance the performance of the classifier, a data selection method based on node similarity was proposed. To evaluate the merit of the proposed methods, a set of experiments were conducted on co-authorship networks of 18 different domains. The result implies the feasibility of achieving significantly high performance for most of the networks using the proposed self training approach.

While conventional crime prediction methods rely on historical crime records and geographical information of the location of interest, we pursue the question of whether a social media context can provide socio-behavior “signals” for a crime prediction problem. The hypothesis is that crowd publicly available data in Twitter may include predictive variables which can indicate changes in crime rates without being only limited to the availability of historical crime records of specific locations. We developed a prediction model for crime trend prediction, where the objective is to employ Twitter content to predict crime rate directions in a prospective time-frame. The model employs content, sentiment, and topics, as the predictive indicators to infer the changes of crime indexes. Since our problem has a sequential order, we propose a temporal topic detection model to infer predictive topics over time. The main challenge of topic detection over time is information evolution, in which data are more related when they are close in time rather than further apart. Our proposed topic detection model builds a dynamic vocabulary to detect emerging topics rather than considering a vocabulary in bulk. We applied our model on data collected from Chicago for crime trend prediction using historical tweets. The results have revealed the correlation between features extracted from the content as content-based features and the crime trends. Moreover, the results indicate the feasibility of our proposed temporal topic detection model in identifying the most predictive features over time compared to a static model without time consideration. We also studied the contribution of socio-economic indexes and temporal features as auxiliary features. The experiment shows the content-based features improve the prediction performance significantly compared to the auxiliary features. Overall, the study provides a deep insight into the correlation between language and crime trends and the impact of social data as an extra resource in providing predictive indicators.

In this paper, an automatic generation of domain-specific stopwords from a large labeled corpus is proposed. In the majority of text mining tasks, stopwords are removed according to a standard stopword list and/or using high and low document frequencies. In this paper, a new approach for stopword extraction, based on the notion of backward filter-level performance and data sparsity index, is proposed. First, based on the proposed model to evaluate the extracted stopwords, we examine high document frequency filtering for stopword reduction. Secondly, a new algorithm for building general and domain-specific stopword lists is proposed. For the method, it is assumed that a set of candidate stopwords must have a minimum information content and prediction capacity that is measured by the performance of a classifier. We show that to avoid obtaining the classifier performance, it can be estimated by the sparsity of the training dataset. Moreover, it is confirmed that even if a given term ranking measure can perform well for the feature selection, the measure is not necessarily efficient for selecting poor features (stopwords). According to the comparative study, the newly devised approach offers more promising results that guarantee a minimum information loss by filtering out most stopwords.

Social media provides increasing opportunities for users to voluntarily share their thoughts and concerns in a large volume of data. While user-generated data from each individual may not provide considerable information, when combined, they include hidden variables, which may convey significant events. In this paper, we pursue the question of whether social media context can provide socio-behavior "signals" for crime prediction. The hypothesis is that crowd publicly available data in social media, in particular Twitter, may include predictive variables, which can indicate the changes in crime rates. We developed a model for crime trend prediction where the objective is to employ Twitter content to identify whether crime rates have dropped or increased for the prospective time frame. We also present a Twitter sampling model to collect historical data to avoid missing data over time. The prediction model was evaluated for different cities in the United States. The experiments revealed the correlation between features extracted from the content and crime rate directions. Overall, the study provides insight into the correlation of social content and crime trends as well as the impact of social data in providing predictive indicators.

When mining user streams from social media, activity gaps are inevitable, which is known as the sparsity of user data. Such sparsity can significantly degrade the performance of a predictive system that relies on time-sensitive user content. To mitigate this issue, conventional approaches generally tend to discard periods with missing data. However, this solution leads to neglecting information generated by other users which, if utilized, could potentially enhance the quality of the predictive model. So the following question arises: is it possible to alleviate the impact of absent data while preserving the available content contributed within the same timespan? Despite the fact that this problem is well-known, it has not been thoroughly studied before. The goal of this work is to find a way of interpolating missing data from user's network and his previous activities. We investigate how different types of user profiles affect overall behavior predictability. Proposed models are evaluated on a case study of a micro-blogging system for the investment community.

The majority of techniques in socio-behavioral modeling tend to consider user-generated content in a bulk, with the assumption that this sort of aggregation would not have any negative impact on overall predictability of the system, which is not necessarily the case. We propose a novel user-centric approach designed specifically to capture most predictive hidden variables that can be discovered in a context of the specific individual. The concept of social filtering closely resembles collaborative filtering with the main difference that none of the considered users intentionally participates in the recommendation process. Its objective is to determine both the subset of best expert users able to reflect a particular social trend of interest and their transformation into feature space used for modeling. We introduce three-step selection procedure that includes activity-and relevance-based filtering and ensemble of expert users, and show that proper choice of expert individuals is critical to prediction quality.

Despite extensive research efforts in stock market predictions using social media networks, there still exists a lack of credible sources of information in such media. This study presents a novel approach to measure the credibility of Twitter users in a domain of interest, namely the stock market. This study suggests a correlation between each user's credibility and the extracted features from each follower network: number of followers, number of stock market-related followers, extracted by a $Cashtag-based approach, ratio of stock market-related followers to the total number of followers, and the number of seed user tweets. The results support the initial hypothesis of this study.

Hierarchical clustering has been well-studied in the community of machine learning. Hierarchical clustering algorithms are deterministic, stable, and do not need a pre-determined number of clusters as input. However, they are not scalable for very large data due to their non-linear complexity. In this paper, a new approach is proposed to reduce the complexity of Hierarchical Clustering, improve the purity of the clustering algorithm, and reduce the chaining factor. The proposed method has the following components: (i) A new combination similarity based on common-neighbours of graph theory is proposed, (ii) In every iteration, instead of calculating the centroids for new clusters, new centroids are estimated from centroids in previous iteration, and (iii) In each iteration, instead of merging only one pair of objects, multiple pairs are merged at the same time. In addition to the proposed combination similarity, four well-known methods including centroid-based, group-based, complete-link, and single-link, have been also implemented. All five methods are tested and evaluated using two metrics: purity and imbalance or chaining factor. We show that our proposed algorithm outperforms other classic methods.

In a polarized society, rhetorical arguments are usually expressed by strong, extreme terms which by themselves carry a positive or negative sentiment in favor or against one side of a social debate or conflict. In a divided society, by detecting extreme terms in a document such as a blog post which is reflecting some political opinion, we are potentially able to automatically detect the sentiment of the text about the polarizing issue. On the other hand, during any social and political conflict in a polarized society, we can observe a shift from mainstream to extreme language or rhetoric. In this paper, a new metric called “language gap” is introduced to estimate the distance between mainstream and rhetoric in a social-political debate. Then we illustrate that there is a correlation between the language shift and social conflicts. In other words, the language shift can be used as a signal for predicting social conflicts in a divided society.

One of the major tasks in text categorization systems is dimensionality reduction, which strongly affects classification performance and scalability. Among dimensionality reduction methods, feature ranking-based feature selection, also known as best

We propose a novel approach to label social media text using significant stock market events (big losses or gains). Since stock events are easily quantifiable using returns from indices or individual stocks, they provide meaningful and automated labels. We extract significant stock movements and collect appropriate pre, post and contemporaneous text from social media sources (for example, tweets from twitter). Subsequently, we assign the respective label (positive or negative) for each tweet. We train a model on this collected set and make predictions for labels of future tweets. We aggregate the net sentiment per each day (amongst other metrics) and show that it holds significant predictive power for subsequent stock market movement. We create successful trading strategies based on this system and find significant returns over other baseline methods.

Feature ranking is widely used in text classification. One problem with feature ranking methods is their non-robust behavior when applied to different data sets. In other words, the feature ranking methods behave differently from one data set to the

In this paper, a new approach to predicting the structure of a social network without any prior knowledge from the social links is proposed. In absence of links among nodes, we assume there are other information resources associated with the nodes which are called node profiles. The task of link prediction and recommendation from text data is to learn similarities between the nodes and then translate pair-wise similarities into social links. In other words, the process is to convert a similarity matrix into an adjacency matrix. In this paper, an alternative approach is proposed. First, hidden topics of node profiles are learned using Latent Dirichlet Allocation. Then, by mapping node-topic and topic-topic relations, a new structure called semi-bipartite graph is generated which is slightly different from regular bipartite graph. Finally, by applying topological metrics such as Katz and short path scores to the new structure, we are able to rank and recommend relevant links to each node. The proposed technique is applied to several co-authorship networks. While most link prediction methods are low precision solutions, the proposed method performs effectively and offers high precision.

Feature ranking is widely employed to deal with high dimensionality in text classification. The main advantage of feature ranking methods is their low cost and simple algorithms. However, they suffer from some drawbacks which cause low performance compared to wrapper approach feature selection methods. In this paper, three major drawbacks of feature ranking methods are discussed. First, we show that feature ranking methods are highly problem dependent. For designing an effective feature ranking method and appropriate ranking threshold, we need background knowledge including the data set characteristics as well as the classifier to be used. Second, the feature ranking methods are univariate functions, while the nature of text classification is multivariate. It means that in these methods, correlation between terms is ignored. Finally, they fail in multiple class problems with unbalanced class distribution because they pay more attention to the simpler and larger classes. In this paper, these drawbacks, especially the last two issues, are experimentally investigated using a set of extensive numerical experiments with several data sets and feature scoring measures.

Data partitioning methods such as bagging and boosting have been extensively used in multiple classifier systems. These methods have shown a great potential for improving classification accuracy. This study is concerned with the analysis of training data distribution and its impact on the performance of multiple classifier systems. In this study, several feature-based and class-based measures are proposed. These measures can be used to estimate statistical characteristics of the training partitions. To assess the effectiveness of different types of training partitions, we generated a large number of disjoint training partitions with distinctive distributions. Then, we empirically assessed these training partitions and their impact on the performance of the system by utilizing the proposed feature-based and class-based measures. We applied the findings of this analysis and developed a new partitioning method called "Clustering, Declustering, and Selection" (CDS). This study presents a comparative analysis of several existing data partitioning methods including our proposed CDS approach.

In this paper, a text classifier framework with a feature ranking scheme is proposed to extract social structures from text data. It is assumed that only a small subset of relations between the individuals in a community is known. With this assumption, the social network extraction is translated into a classification problem. The relations between two individuals are represented by merging their document vectors and the given relations are used as labels of training data. By this transformation, a text classifier such as Rocchio is used for learning the unknown relations. We show that there is a link between the intrinsic sparsity of social networks and class imbalance. Furthermore, we show that feature ranking methods usually fail in problem with unbalanced data. In order to deal with this deficiency and re-balance the unbalanced social data, a local feature ranking method, which is called reverse discrimination, is proposed.

Automatic building of social networks requires extracting pair-wise relations of the individuals. In this paper, supervised learning of social networks from a set of documents is proposed. Given a small subset of relations between the individuals, the problem of learning social network is translated into a text classification problem. Relation between each pair of individuals is represented by a vector of words produced from merging all documents associated with these two individuals. The known relation is used as a label for the relation vector. The merged documents and their given labels, are used as training data. By this transformation, a text classifier such as SVM can be used for learning the unknown relations. We show that there is a link between the intrinsic sparsity of social networks and class distribution imbalance of the training data. In order to re-balance the unbalanced training data, a multiple resampling method, including undersampling of the majority and oversampling of the minority class, is employed. The proposed framework is applied to a Friend Of A Friend (FOAF) data set and evaluated by the macro-averaged F-measure.

Feature ranking is one of the dimensionality reduction methods. Because of its simplicity and low cost, it is widely used in text classification. One problem with feature ranking methods is their non-robust behavior when applied to different data sets. In other words, the feature ranking methods behave differently from one data set to the other. The problem is more complex when we consider that the performance of feature ranking methods is different when being used by different classifiers. In this paper, a new method based on combining feature rankings is proposed to find the best features among a set of feature rankings. Four preferential voting method are employed to combine feature rankings obtained by eight well-known ranking measures. According to the results, combining methods can offer reliable results that are very close to the best solution without the need to use a classifier. The proposed method is applied to the text classification problem and evaluated on three well-known data sets using SVM classifier.

An automatic taxonomy extraction algorithm is proposed. Given a set of terms or terminology related to a subject domain, the proposed approach uses Google page count to estimate the dependency links between the terms. A taxonomic link is an asymmetric relation between two concepts. In order to extract these directed links, neither mutual information nor normalized Google distance can be employed. Using the new measure of information theoretic inclusion index, term dependency matrix, which represents the pair-wise dependencies, is obtained. Next, using a proposed algorithm, the dependency matrix is converted into an adjacency matrix, representing the taxonomy tree. In order to evaluate the performance of the proposed approach, it is applied to several domains for taxonomy extraction.

Automatic generation of a social network requires extracting pair-wise relations of the individuals. In this research, learning social network from incomplete relationship data is proposed. It is assumed that only a small subset of relations between the individuals is known. With this assumption, the social network extraction is translated into a text classification problem. The relations between two individuals are modeled by merging their document vectors and the given relations are used as labels of training data. By this transformation, a text classifier such as SVM is used for learning the unknown relations. We show that there is a link between the intrinsic sparsity of social networks and class distribution imbalance of the training data. In order to re-balance the unbalanced training data, a minority class down-sampling strategy is employed. The proposed framework is applied to a true FOAF (friend of a friend) database and evaluated by the macro-averaged F-measure