We present a platform for automatic assessment of technical data science skills (hard skills) and competencies that help to apply those technical skills in practice (soft skills). The platform serves so-called assessment platform tasks that resemble data-focused tasks typical for online data science competitions. Actually, these tasks are designed based on international conference competitions that we have been organizing on our competition platform knowledgepit.ai. The main idea relies on our observation that the given person’s behavior during a competition (dynamics of submitting solutions, activity on competition forum, etc.) can be correlated with his/her soft competencies. Accordingly, our goal is to translate the specifics of international conference competitions into a framework that investigates the behaviors of data science job candidates, employees who want to improve their work in data science projects, students who wish to build their future professional careers on AI solutions, etc. We claim that such behaviors – if properly measured based on solving our assessment platform tasks – can effectively indicate both hard and soft types of competencies.

We summarize the results of the IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty – a data science competition organized at the knowledgepit.ai platform in association with the IEEE BigData 2024 conference. We describe the competition goal and tie it to existing research on human-computer interaction, focusing on task difficulty estimation and aligning human and AI behavior. We explain how we acquired and processed the data, separately for training and testing datasets. We review submitted solutions and evaluate their performance by comparing them to a simple benchmark. We explain how the achieved rating differences translate to user experience when solving chess puzzles. We further explore the concept of chess puzzle difficulty by replicating competition results with puzzle ratings obtained using chess bots only. We conclude with a summary of our findings and directions for future studies, as well as an invitation to the next edition of the competition in 2025.

We summarize the results of the IEEE BigData 2023 Cup: Object Recognition with Muon Tomography using Cosmic Rays - a data mining competition organized at the KnowledgePit.ai platform in association with the IEEE BigData 2023 conference. We describe the challenge at the heart of the competition task, as well as the data acquisition and preparation steps. We present the entire process of preparing experiments and subsequent data analysis for the purpose of recognizing X-rayed objects using muon tomography techniques. We conclude this analysis by presenting the baseline as well as the winning solution of the object segmentation algorithms for the research space reconstruction and object classification.

Subject matter experts (SMEs) often rely on attribute importance rankings to verify machine learning models, acquire insights into their outcomes, and gain a deeper understanding of the investigated phenomena. To further increase their usefulness, we introduce a new approach to the evaluation of attribute rankings produced by any machine learning method . As a real-world case study , we investigate the attribute importance scores produced using XGBoost and decision bireducts on the data gathered by an HR company, where the goal is to predict the willingness of candidates to change their job. For this task, XGBoost delivers accurate models but fails to identify many attributes that are important to SMEs. In comparison, decision bireducts lead to models that are easier to interpret and explore the data with a higher focus on the diversity of attributes. The ensembles of decision bireducts deliver comparable accuracy and their associated attribute rankings are more insightful than those of XGBoost.

Regression problems frequently occur in the surrounding world, therefore are unavoidable in real-world applications. However, to obtain a model with desired generalization performance, a vast amount of labels usually is required. In many scenarios, obtaining unlabelled data is relatively inexpensive, therefore active learning approaches may be used to reduce the needed annotation effort. Most uncertainty-based regression active learning algorithms use variance estimation of model predictions to choose informative samples. Those algorithms do not incorporate knowledge about the data distribution for the given task. In this paper, we propose a novel algorithm to incorporate information about data distribution and combine it with variance estimation as an informativeness function. Experiments conducted on four data sets show that the proposed approach outperforms standard variance-based sampling by a margin, and indicate its robustness.

We summarize the results of IEEE BigData 2022 Cup: Privacy-preserving Matching of Encrypted Images - a data mining challenge organized at the KnowledgePit platform in association with the IEEE BigData 2022 conference. We describe the challenge in the hearth of the competition task, as well as the data acquisition and preparation steps. We also provide a brief overview of the top-performing solutions submitted by participants. Finally, we present results of the post-competition data analysis, in which we consider the similarity of solutions submitted by various teams in terms of their errors on the test data. We conclude this analysis by discussion of the significance and impact of the competition results on the underlying problem of constructing effective and efficient anonymization algorithms for monitoring in the DOOH advertising industry.

We summarize the results of IEEE BigData 2021 Cup: Predicting Victories in Video Games - a data mining challenge organized at the KnowledgePit platform in association with the IEEE BigData 2021 conference. We describe the competition task, as well as the data acquisition and preprocessing steps. We also provide a brief overview of the top-performing solutions submitted by participants. Finally, we present results of the post-competition data analysis, in which we consider the similarity of solutions submitted by various teams in terms of their errors on the test data. We conclude this analysis by demonstrating a method for constructing an ensemble of submitted solutions. Such an ensemble performs better than any of the individual solutions submitted during the competition.

We summarize IEEE Big Data Cup: Predicting Escalations in Customer Support - a data mining competition organized jointly by companies Information Builders and QED Software at the KnowledgePit platform, in the frame of the 2020 IEEE International Conference on Big Data. We discuss the motivation for organizing this event and highlight the factors that make it such a challenging topic. We describe the data provided to participants and formulate the competition task. We also provide an overview of competition results with a detailed analysis of a few selected solutions. Finally, we present a novel functionality of the KnowledgePit platform - an analytic module that allows organizers to investigate selected solutions using a convenient GUI and provides in-depth insights about their quality.

“IEEE BigData 2019 Cup: Suspicious Network Event Recognition” was a data mining competition organized jointly by companies Security On-Demand and QED Software at the KnowledgePit online platform, in association with the IEEE BigData 2019 conference. The scope of this challenge referred to the notions of cybersecurity analytics and network alert evaluation. In this paper, we summarize the results of our competition. We explain how data sets had been prepared before it was possible to make them available to competition participants. We describe the baseline scoring models that we designed as a reference for participants, and we demonstrate how critical for their performance was the aspect of appropriate feature engineering. We also discuss the results of experiments conducted to verify the (un)suitability of deep recurrent neural networks in this particular case. In some sense, we show that there are no “perfect” machine learning approaches that could be applied equally successfully to every data science undertaking.

We consider a problem of learning from compact representations of images for a purpose of object recognition and content-based image retrieval. We discuss a motivation for using compressed images in those tasks and indicate exemplary applications related to analysis on the data from satellites. Finally, we show some preliminary results of experiments conducted to demonstrate the impact of the image data granulation on the quality of classification. We empirically compare the performance of prediction models trained on original images, images compressed using autoencoders, and on images whose quality was lowered in order to reduce their size.

In this article, we describe the SENSEI system. It helps players to improve their skills in popular eSports games. We discuss the main goals of the system and explain the associated challenges. We also present its conceptual architecture which aims at enabling full automation of the data acquisition and analytic processes. The system is expected to provide in-depth analytics of players' performance and give practical advice regarding possible improvements. Thus its architecture allows players to provide feedback and manually label important concepts. Finally, we discuss our first case study - an advisory system for popular collectible card video games.

Information granulation plays an important role in the process of scaling up modern machine learning and knowledge discovery algorithms. By employing compact descriptions of granules - whereby granules are defined as collections of original data elements gathered together by means of their similarity, proximity or functionality - one can drastically accelerate computations and, moreover, make the results of those computations more meaningful for domain experts. In this paper, we summarize some of the feature space granulation approaches introduced by now. We discuss the meaning of similarity, proximity and functionality while considering the granules of physically existing or potentially derivable attributes. We also show several examples of utilization of the granulation structures defined over the feature spaces in the feature selection algorithms. As a case study, we consider the algorithms developed within the theory of rough sets, aimed at finding irreducible subsets of attributes that are sufficient to distinguish between the cases belonging to different target decision classes.