Chess is a complex logical game involving ongoing strategic forward planning and evaluation. Solving chess puzzles is one of the most common ways of training and developing chess skills. It involves continuing the game from a certain initial chessboard state against a real or AI opponent until defeat or a significant advantage is achieved. To ensure solving chess puzzles is efficient and engaging for the real player, it is important to understand the difficulty of the puzzle and match it with the skill of the solver. Accurate and fast assessment of puzzle difficulty is therefore a critical problem that online chess platforms need to solve at scale to optimally match many real players with adequate puzzles. To solve this challenge, we propose a chess knowledge-agnostic strategy to predict puzzle difficulty based solely on the moves of the players against their opponents trying to solve the puzzles. Specifically designed deep convolutional neural networks (CNN) were deployed as supervised learning predictors, fed with player moves represented as multichannel chessboard images. Extensive testing of our model with almost 4 million training examples against Glicko-2 evaluated puzzle difficulty ratings—considered as ground truth—resulted in good predictive performance. This was acknowledged by our runner-up result as the 7th place in the IEEE Big Data 2024 Cup and highlighted the capability of fast puzzle difficulty prediction based only on players’ moves as evidence, with no prior chess knowledge nor utilization of computationally expensive chess engines.

Causal Inference has wide applications in various areas such as E-commerce and precision medicine, and its performance heavily relies on the accurate estimation of the Individual Treatment Effect (ITE). Conventionally, ITE is predicted by modeling the treated and control response functions separately in their individual sample spaces. However, such an approach usually encounters two issues in practice, i.e. divergent distribution between treated and control groups due to treatment bias, and significant sample imbalance of their population sizes. This paper proposes Deep Entire Space Cross Networks (DESCN) to model treatment effects from an end-to-end perspective. DESCN captures the integrated information of the treatment propensity, the response, and the hidden treatment effect through a cross network in a multi-task learning manner. Our method jointly learns the treatment and response functions in the entire sample space to avoid treatment bias and employs an intermediate pseudo treatment effect prediction network to relieve sample imbalance. Extensive experiments are conducted on a synthetic dataset and a large-scaled production dataset from the E-commerce voucher distribution business. The results indicate that DESCN can successfully enhance the accuracy of ITE estimation and improve the uplift ranking performance. A sample of the production dataset and the source code are released to facilitate future research in the community, which is, to the best of our knowledge, the first large-scale public biased treatment dataset for causal inference.

An accurate evaluation of player-win likelihoods during online games is critical for controlling the attractiveness and immersion of the gameplay, especially if played against an AI bot. Predicting game victories is a very challenging problem heavily depends on the complexity and the stage of the game. With the multitude of ways and formats that the players, the gameplay, individual state and history can be encoded and utilized by machine learning (ML) models, it is fascinating to take part in the IEEE Big Data 2021 Cup and explore which data representations, feature engineering methods and prediction models work better than others and what levels of predictability they can achieve in the specific use case of predicting victories in Tactical Troops: Anthracite Shift online computer game. Our explorations throughout the game and a high 4th place in the Cup’s final indicate that a careful and comprehensive feature engineering methodology numerically capturing the last state of the game paired with the variants of the gradient boosting models offer a robust and competitive solution for this challenge.

At anytime in a tactics game, given the current status of the game players, it’s always interesting to predict who is going to win the game in the end. This is the challenge defined in the IEEE BigData Cup 2021, where the objective is to predict the likelihood of winning the game, Tactical Troops: Anthracite Shift, given information of the players as well as logged data of the game at the time the prediction needs to be made. To address this challenge, we have developed a hybrid model that combines a general model trained on all the data and two specific models trained on particular data of two distinct game modes. Our implementation was realized by LightGBM, one of the most popular tree-based libraries implementing the state-of-the-art Extreme Gradient Boosting algorithm. In addition to combined models, a thoroughly feature engineering and feature selection together with a special handling of missing values and categorical features, a careful hyper-parameter tuning, and a robust cross-validation strategy to avoid overfitting issues helped us win the 2ndplace in the competition.

The rapid expansion of Internet of Things (IoT) quickly evolves towards a connected network of AI-enabled, smart, multi-sensory devices that generate, consume and exchange enormous amounts of data. Stimulated by the omnipresent cloud-services and ever widening data bandwidth they aspire to control every aspect of our lives from transportation to our health, significantly increasing our reliance on the web-based services and their security. Already millions of cyber security alerts are generated every day and trigger increasingly costly investigations of the Security Operations Centres (SOC). In order to make their operation more efficient security warnings need to be reliably detected and classified based on various levels of severity, scale of potential damage or an ability to defend. We have responded to this challenge in the context of IEEE BigData Cup 2019 focused on predicting cyber security threats that require attention based on detailed logs of network activity leading to the security alert. We have developed a hybrid supervised learning ensemble model combining several state-of-the-art Extreme Gradient Boosting algorithms. Specifically, xGBoost and LightGBM model versions have been built on separate sets of features extracted from the raw logs of network events preceding the alerts and then synergically aggregated. Models' diversity imposed by algorithmic differences, complementary feature subsets, and individually optimized hyperparameters, combined with robust stratified cross-validation scheme resulted with the best true alerts detection rate yielding the AUC score in excess of 0.93, that outperformed all other 248 competing teams.

Sequential, predominantly temporal nature of the vast amounts of big data released every day from many different sources could potentially be linked, aligned along the time and deliver new evidence for the next generation predictive systems or knowledge discovery engines. However, big data owners are reluctant to share their data due to legally binding privacy and identity protection concerns, thereby posing a major hurdle preventing shared exploitation of big data on a massive scale. Data anonymization is expected to solve this problem, yet the current approaches are limited predominantly to univariate time series generalized by aggregation or clustering to eliminate identifiable uniqueness of individual data points or patterns. For multivariate time series, uniqueness among of the combination of values or patterns across multiple dimensions is much harder to eliminate due the to exponentially growing number of unique configurations of point values across multiple dimensions. Our method implements linearly scalable asynchronous summarization of multivariate time series independently at every dimension. As a result the series retain only a small subset of defining points at different times along multiple dimensions effectively breaking up the multivariate time series into a collection of summarized univariate time series that are perturbed from the original series in terms of actual points and pattern shapes. Current implementation of the anonymizing summarization involves shape preserving greedy elimination and aggregation that supports parallel cluster processing for big data implementation.