In the dynamic field of financial analytics, the ability to predict stock market trends is crucial for effective trading strategies, which is the task for FedCSIS 2024 Data Science Challenge: Predicting Stock Trends.This paper presents a comprehensive study on the use of hybrid gradient boosting models, incorporating both classification and regression approaches, to forecast stock trends across different sectors of the S&P 500.Utilizing a rich dataset comprising key financial indicators for 300 companies over a decade, our research aims to unravel the complexities of sector-specific trend predictions.The model leverages 58 financial indicators per company, along with their annual change metrics, to predict the future stock movements.In the preliminary phase of the competition, our hybrid model demonstrated promising results, achieving the lowest weighted error of 0.5941 among competitors.However, despite the initial success, the final phase of the model evaluation revealed a significant performance decline with the error rising above 0.84.This discrepancy highlights potential issues in model stability and preliminary performance when transitioning from a controlled to a truly unseen testing environment.This work not only underscores the complexities of predictive modeling in finance but also sets the stage for future research into creating more resilient AI-driven trading systems.

Monitoring systems have become more and more popular. A typical example is a smart home monitoring system which provides peace of mind for people staying far away from their homes. This system can be deployed for different purposes, such as to take care elderly staying at home alone or to provide anti-theft when nobody is at home. In this type of monitoring systems, there is always a problem with protecting the privacy and security of captured images. This problem is often addressed by either scrambling or encoding techniques that add noise in or to distort the images. In the IEEE Big Data 2022 Cup, the challenge is to detect and perform matching between an original image and its scrambled or encoded version. Generally, if we are able to train a machine learning model that would correctly identify the matching, it will be proof that the applied scrambling or encoding technique is not strong enough to protect data and hence it needs to be strengthened further. In this paper, we present an auto-encoder architecture that allows us to demystify scrambled or encoded images as long as the scrambling or encoding techniques are deterministic. Source code is publicly available1.

A common business practice for transportation forwarders is to bid for shipping contracts at the transport or freight exchanges.Based on the detailed contract requirements they try to estimate the total expected cost of its execution and accordingly bid with the fixed price in advance for delivering such shipping service at the prescribed specification and schedule.The capability to accurately predict the cost of contract execution is the critical factor deciding about the profitability of offered shipping services as well as the amount of business drawn from freight exchanges.However, given highly volatile nature of the transport services ecosystem, it is difficult to simultaneously account for countless dynamically changing factors like fuel prices, currency exchange rates, temporal and spatial multitude of routing and implied traffic risks, the properties of cargo and shipping vehicles etc., which leads to big cost under-or overestimation resulting with loss-making contracts or equally painful missed revenue opportunities.In the context of FedCSIS 2022 data mining competition we propose an accurate and robust predictor of the cost of forwarding contracts built upon the detailed contract data using the ensemble of the state-of-theart gradient boosting-based regression models.Our established feature engineering framework combined with deep parametric optimization of the individual models and multi-faceted diversification techniques guiding hybrid final model ensembles were instrumental to outperform all the competitive predictors and win the FedCSIS 2022 contest.

Predicting the cost of forwarding contracts is a typical problem that logistics companies need to solve in order to optimize their business for a better profit.This is the challenge defined in the FedCSIS 2022 Competition where a fiveyear history of contract data and their delivery routes from a large Polish logistics company are provided to train a Machine Learning model.In addition to the contract data, historical wholesale fuel prices and euro exchange rates at the contract time are also provided.To address this challenge, we first designed a basic solution where we focused on feature engineering to find good impact features for the model.After that, the same set of features were used to train two different models: one using XGBoost and the other using LightGBM.The average predictions of the two boosting models were then used as the predictions for the next post-processing step.Finally, in the post-processing step, we designed and trained a simple linear regression model to capture the average monthly changes of the contract cost, given the changes of the fuel prices and euro exchange rates.These captured changes were used to post-process (adjust) the predictions in the previous step to address the issue that treebased models could not predict the value that they did not see before.While the basic solution with careful feature selection gave us a place in the top-5, our post-processing strategy in the last step helped us win the 3 rd prize in the competition.

An accurate evaluation of player-win likelihoods during online games is critical for controlling the attractiveness and immersion of the gameplay, especially if played against an AI bot. Predicting game victories is a very challenging problem heavily depends on the complexity and the stage of the game. With the multitude of ways and formats that the players, the gameplay, individual state and history can be encoded and utilized by machine learning (ML) models, it is fascinating to take part in the IEEE Big Data 2021 Cup and explore which data representations, feature engineering methods and prediction models work better than others and what levels of predictability they can achieve in the specific use case of predicting victories in Tactical Troops: Anthracite Shift online computer game. Our explorations throughout the game and a high 4th place in the Cup’s final indicate that a careful and comprehensive feature engineering methodology numerically capturing the last state of the game paired with the variants of the gradient boosting models offer a robust and competitive solution for this challenge.

At anytime in a tactics game, given the current status of the game players, it’s always interesting to predict who is going to win the game in the end. This is the challenge defined in the IEEE BigData Cup 2021, where the objective is to predict the likelihood of winning the game, Tactical Troops: Anthracite Shift, given information of the players as well as logged data of the game at the time the prediction needs to be made. To address this challenge, we have developed a hybrid model that combines a general model trained on all the data and two specific models trained on particular data of two distinct game modes. Our implementation was realized by LightGBM, one of the most popular tree-based libraries implementing the state-of-the-art Extreme Gradient Boosting algorithm. In addition to combined models, a thoroughly feature engineering and feature selection together with a special handling of missing values and categorical features, a careful hyper-parameter tuning, and a robust cross-validation strategy to avoid overfitting issues helped us win the 2ndplace in the competition.

Deep convolutional neural networks revolutionized the area of automated objects detection from images.Can the same be achieved in the domain of time series forecasting?Can one build a universal deep network that once trained on the past would be able to deliver accurate predictions reaching deep into the future for any even most diverse time series?This work is a first step in an attempt to address such a challenge in the context of a FEDCSIS'2020 Competition dedicated to network device workload prediction based on their historical time series data.We have developed and pre-trained a universal 3-layer bi-directional Long-Short-Term-Memory (LSTM) regression network that reported the most accurate hourly predictions of the weekly workload time series from the thousands of different network devices with diverse shape and seasonality profiles.We will also show how intuitive human-led post-processing of the raw LSTM predictions could easily destroy the generalization abilities of such prediction model.

The rapid expansion of Internet of Things (IoT) quickly evolves towards a connected network of AI-enabled, smart, multi-sensory devices that generate, consume and exchange enormous amounts of data. Stimulated by the omnipresent cloud-services and ever widening data bandwidth they aspire to control every aspect of our lives from transportation to our health, significantly increasing our reliance on the web-based services and their security. Already millions of cyber security alerts are generated every day and trigger increasingly costly investigations of the Security Operations Centres (SOC). In order to make their operation more efficient security warnings need to be reliably detected and classified based on various levels of severity, scale of potential damage or an ability to defend. We have responded to this challenge in the context of IEEE BigData Cup 2019 focused on predicting cyber security threats that require attention based on detailed logs of network activity leading to the security alert. We have developed a hybrid supervised learning ensemble model combining several state-of-the-art Extreme Gradient Boosting algorithms. Specifically, xGBoost and LightGBM model versions have been built on separate sets of features extracted from the raw logs of network events preceding the alerts and then synergically aggregated. Models' diversity imposed by algorithmic differences, complementary feature subsets, and individually optimized hyperparameters, combined with robust stratified cross-validation scheme resulted with the best true alerts detection rate yielding the AUC score in excess of 0.93, that outperformed all other 248 competing teams.

Support Vector Regression (SVR) as a supervised machine learning algorithm have gained popularity in various fields.However, the quadratic complexity of the SVR in the number of training examples prevents it from many practical applications with large training datasets.This paper aims to explore efficient ways that maximize prediction accuracy of the SVR at the minimum number of training examples.For this purpose, a clustered greedy strategy and a Genetic Algorithm (GA) based approach are proposed for optimal subset selection.The performance of the developed methods has been illustrated in the context of Clash Royale Challenge 2019, concerned with decks' win rate prediction.The training dataset with 100,000 examples were reduced to hundreds, which were fed to SVR training to maximize model prediction performance measured in validation R 2 score.Our approach achieved the second highest score among over hundred participating teams in this challenge.

Support Vector Regression (SVR) is a powerful supervised machine learning model especially well suited to the normalized or binarized data.However, its quadratic complexity in the number of training examples eliminates it from training on large datasets, especially high dimensional with frequent retraining requirement.We propose a simple two-stage greedy selection of training data for SVR to maximize its validation set accuracy at the minimum number of training examples and illustrate the performance of such strategy in the context of Clash Royale Challenge 2019, concerned with efficient decks' win rate prediction.Hundreds of thousands of labelled data examples were reduced to hundreds, optimized SVR was trained on to maximize the validation R 2 score.The proposed model scored the first place in the Cash Royale 2019 challenge, outperforming over hundred of competitive teams from around the world.

Designing a robust and adaptable Artificial Intelligence (AI) opponent in a computer game would ensure the game continues to challenge, immerse and excite the players at any stage.The outcomes of card based games such as "Heartstone: Heroes of Warcraft", aside the player skills, heavily depend on the initial composition of player card decks.To evaluate this impact we have developed a new robust regression network in a context of the AAIA Data Mining Competition 2018, which tries to predict the average win-rates of the specific combinations of bot-player and card decks.Our network is composed of 2 levels: the entry level with an array of finely optimized state of the art regression models including Extreme Learning Machines (ELM), Extreme Gradient Boosted decision tree (XGBOOST), and Least Absolute Shrinkage and Selection Operator (LASSO) regression trained via supervised learning on the labeled training dataset; and just a single ELM at the 2 nd level installed to learn to correct the predictions from the 1 st level.The final solution received the root of the mean squared error (RMSE) of just 5.65% and scored the 2 nd place in AAIA'2018 competition.This paper also presents two other runner-up models receiving RMSE of 5.7% and 5.86%, scoring the 4 th and the 6 th place respectively.

Success of many computer games depends on designing a robust and adaptable AI opponent that would ensure the games continue to challenge, immerse and excite the players at any stage.The outcomes of card based games like "Heartstone: Heros of Warcraft", aside the player skills heavily depend on the initial composition of player card decks.To evaluate this impact we have developed an ensemble prediction model that tries to predict the average win-rates of the specific combination of bot-player and card decks.Our ensemble model consists of three sub-models: two Logistic Regression models and one Deep Learning model.The models are trained with both provided data and additional data about the cards, their health, attack power and cost.To avoid overfitting, we employ a trick to generate predictions for all possible combinations of opponent players and decks and obtain the result as the average of all these predictions.

This paper introduces an ensemble model that solves the binary classification problem by incorporating the basic Logistic Regression with the two recent advanced paradigms: extreme gradient boosted decision trees (xgboost) and deep learning.To obtain the best result when integrating sub-models, we introduce a solution to split and select sets of features for the sub-model training.In addition to the ensemble model, we propose a flexible robust and highly scalable new scheme for building a composite classifier that tries to simultaneously implement multiple layers of model decomposition and outputs aggregation to maximally reduce both bias and variance (spread) components of classification errors.We demonstrate the power of our ensemble model to solve the problem of predicting the outcome of Hearthstone, a turn-based computer game, based on game state information.Excellent predictive performance of our model has been acknowledged by the second place scored in the final ranking among 188 competing teams.

This article introduces a dynamic cloud-based marketplace of near-realtime human sensing data (MARSA) for different stakeholders to sell and buy near-realtime data. MARSA is designed for environments where information technology (IT) infrastructures are not well developed but the need to gather and sell near-realtime data is great. To this end, we present techniques for selecting data types and managing data contracts based on different cost models, quality of data, and data rights. We design our MARSA platform by leveraging different data transferring solutions to enable an open and scalable communication mechanism between sellers (data providers) and buyers (data consumers). To evaluate MARSA, we carry out several experiments with the near-realtime transportation data provided by people in Ho Chi Minh City, Vietnam, and simulated scenarios in multicloud environments.

Smart grid is a technological innovation that improves efficiency, reliability, economics, and sustainability of electricity services. It plays a crucial role in modern energy infrastructure. The main challenges of smart grids, however, are how to manage different types of front-end intelligent devices such as power assets and smart meters efficiently; and how to process a huge amount of data received from these devices. Cloud computing, a technology that provides computational resources on demands, is a good candidate to address these challenges since it has several good properties such as energy saving, cost saving, agility, scalability, and flexibility. In this paper, we propose a secure cloud computing based framework for big data information management in smart grids, which we call “Smart-Frame.” The main idea of our framework is to build a hierarchical structure of cloud computing centers to provide different types of computing services for information management and big data analysis. In addition to this structural framework, we present a security solution based on identity-based encryption, signature and proxy re-encryption to address critical security issues of the proposed framework.

We put forward a notion of forward security for stateful public-key encryption against state exposure and chosen ciphertext attack. This new notion is important in mobile applications in which small devices that perform ‘stateful encryptions’ are vulnerable to attacks which can result in the compromise of internal states. We precisely formulate a security definition and propose two efficient schemes which are provably secure under standard computational assumptions.

We propose an efficient generic construction of forward-secure identity-based signature (FSIBS) that ensures unforgeability of past signatures in spite of the exposure of the current signing key. Our construction, supported by formal security analysis, brings about concrete FSIBS schemes which are more efficient than existing schemes in the literature. Especially, one of our instantiations of FSIBS based on discrete-log primitive turns out to be the most efficient among existing ones. As a secondary contribution, we refine the definition of security of FSIBS in such a way that users in the system can freely specify time periods over which their signing keys evolve.

Cloud computing based Data-as-a-Service (DaaS) has become popular. Several data assets have been released in DaaSes across different cloud platforms. Nevertheless, there are no well-defined ways to describe DaaSes and their associated data assets. On the one hand, existing DaaS providers simply use HTML documents to describe their service. This simple way of service description requires user to manually perform service lookup by reading the HTML documents to understand DaaSes as well as their provided data assets. On the other hand, existing service description techniques are not suitable for describing DaaSes because they consider only service information. The lack of well-structured/linked model to describe DaaSes hinders the automatic service lookup for DaaSes and the integration of DaaSes into data composition and analytic tools. In this paper, we propose DEMODS, a Description Model for DaaS, which introduces a general linked model to cover all basic information of a DaaS. Besides the basic DaaS description model, we also introduce an extended model that integrates existing work in describing quality of data, data and service contract, data dependency, and Quality of Service (QoS). We present a mechanism to incorporate DEMODS into both new and existing DaaSes. Finally, a prototype of DEMODS has been developed to evaluate the effectiveness of the proposed model.

Peer-to-Peer (P2P) systems are generally known as open and free systems since resources are often contributed from voluntary participants. Their limitation, however, is that they only support simple resource sharing such as file (storage) sharing and bandwidth sharing. On the other hand, based on virtualization techniques, cloud computing provides a more flexible mechanism to share all types of resources with no limitations. Nevertheless, the use of cloud computing usually requires a fee and is often locked-in to services of cloud computing providers (i.e., vendor lock-in). In this paper, we aim to combine P2P and cloud computing technologies to design and develop PC2, an open and free cloud computing platform that allows users to participate in any time to contribute as well as to consume sharing resources. While this combination inherits good advantages of these two technologies, it also avoid their disadvantages. As a proof-of-concept, we have developed a prototype for PC2based on Eucalyptus, the most popular open source for building cloud computing platforms.

Cloud computing technology has become popular of late due to several advantages it has compared to traditional computing models. For example, cloud computing helps users to save a significant amount of cost that is related to infrastructure investment, management, and maintenance. Furthermore, since cloud computing offers agility, flexibility, and scalability, applications could be deployed, developed, and scaled quickly and easily on the cloud. Nevertheless, while it is not difficult to develop a new application on the cloud, there is a question of whether legacy applications could be moved to the cloud. Knowing the answer for this question is very important to users, who have stable applications in traditional computing platforms, but want to move these applications to the cloud to leverage the advantages of cloud computing. In this paper, we will analyze features of cloud computing services to introduce an answer for this question. In particular, our answer will focus on the practicability and methodology of migrating legacy applications to the cloud. As a proof-of-concept, we support our answer with the migration of three existing applications being used at British Telecom to Google Apps Engine: one is a Java-based application while the two others are Python-based applications. Among the two Python-based applications, one requires a migration of its database in addition to the migration of the application itself.

When users submit new queries to a distributed stream processing system (DSPS), a query planner must allocate physical resources, such as CPU cores, memory and network bandwidth, from a set of hosts to queries. Allocation decisions must provide the correct mix of resources required by queries, while achieving an efficient overall allocation to scale in the number of admitted queries. By exploiting overlap between queries and reusing partial results, a query planner can conserve resources but has to carry out more complex planning decisions. In this paper, we describe SQPR, a query planner that targets DSPSs in data centre environments with heterogeneous resources. SQPR models query admission, allocation and reuse as a single constrained optimisation problem and solves an approximate version to achieve scalability. It prevents individual resources from becoming bottlenecks by re-planning past allocation decisions and supports different allocation objectives. As our experimental evaluation in comparison with a state-of-the-art planner shows SQPR makes efficient resource allocation decisions, even with a high utilisation of resources, with acceptable overheads.

Traditionally, distributed databases assume that the (small) set of nodes participating in a query is known apriori, the data is well placed, and the statistics are readily available. However, these assumptions are no longer valid in a peer-based database management system (PDBMS). As such, it is a challenge to process and optimize queries in a PDBMS. In this paper, we present our distributed solution to this problem for multi-way join queries. Our approach first processes a multi-way join query based on an initial query evaluation plan (generated using statistical data that may be obsolete or inaccurate); as the query is being processed, statistics obtained on-the-fly are used to (continuously) refine the current plan dynamically into a more effective one. We have conducted an extensive performance study which shows that our adaptive query processing strategy can reduce the network traffic significantly.

Attack signature validation plays a key role in intrusion detection and prevention technologies. Usually, when new attacks, particularly worms, appear, security software analyzes and generates signatures for these attacks. Since inaccurate signatures may block legitimate traffic that is similar to the attack traffic (false positives), security software is reluctant to deploy new signatures without extensive testing. The testing procedure, however, can be time consuming, resulting in significant delays (hours or even days) in signature dissemination. To alleviate this problem, in this paper, we propose a novel architecture based on P2P technology for fast content signature validation. The basic idea is to collect and store recent network traffic at peers participating in the system in advance and use it to validate new signatures. Since the amount of traffic that needs to be checked against is huge, we also propose a high-performance validation algorithm over stored traffic data. Experimental results show that our proposed system can validate candidate attack signatures and determine potential false positives rates in just a few seconds.

Over the past few years, peer-to-peer (P2P) systems have rapidly grown in popularity and have become a dominant means for sharing resources. In these systems, load balancing is a key challenge because nodes are often heterogeneous. While several load-balancing schemes have been proposed in the literature, these solutions are typically ad hoc, heuristic based, and localized. In this paper, we present a general framework, HiGLOB, for global load balancing in structured P2P systems. Each node in HiGLOB has two key components: 1) a histogram manager maintains a histogram that reflects a global view of the distribution of the load in the system, and 2) a load-balancing manager that redistributes the load whenever the node becomes overloaded or underloaded. We exploit the routing metadata to partition the P2P network into nonoverlapping regions corresponding to the histogram buckets. We propose mechanisms to keep the cost of constructing and maintaining the histograms low. We further show that our scheme can control and bound the amount of load imbalance across the system. Finally, we demonstrate the effectiveness of HiGLOB by instantiating it over three existing structured P2P systems: Skip Graph, BATON, and Chord. Our experimental results indicate that our approach works well in practice.

We introduce COSTA, for content-based search using term aggregation. Besides advantages shared with other P2P-based information retrieval systems, the system has several characteristics that distinguish itself from other systems: First, an adaptive indexing scheme which can dynamically identify important terms is used. Important terms are indexed in a chord-like ring, while other terms are aggregated in a balanced tree. We argue that this architecture is more flexible and suitable for term indexing than DHT-based methods. Furthermore, this structure allows to eliminate the requirement of maintaining global knowledge, and hence we can avoid the difficulty in maintaining such knowledge. Term aggregation is useful not only for performance enhancement, but also for improving the quality of search, by using of the term statistics information obtained via the aggregation. Traditional IR techniques such as query expansion can be utilized based on the information. Therefore, COSTA finely integrates distributed indexing with information retrieval. Advanced techniques, such as node clustering, caching and workload balance, are employed. We show that more existing optimization techniques can be adopted for further improvement of the system's performance.

While database management systems offer a comprehensive solution to data storage, they require deep knowledge of the schema, as well as the data manipulation language, in order to perform effective retrieval. Since these requirements pose a problem to lay or occasional users, several methods incorporate keyword search (KS) into relational databases. However, most of the existing techniques focus on querying a single DBMS. On the other hand, the proliferation of distributed databases in several conventional and emerging applications necessitates the support for keyword-based data sharing and querying over multiple DMBSs. In order to avoid the high cost of searching in numerous, potentially irrelevant, databases in such systems, we propose G-KS, a novel method for selecting the top-K candidates based on their potential to contain results for a given query. G-KSsummarizes each database by a keyword relationship graph, where nodes represent terms and edges describe relationships between them. Keyword relationship graphs are utilized for computing the similarity between each database and a KS query, so that, during query processing, only the most promising databases are searched. An extensive experimental evaluation demonstrates that G-KS outperforms the current state-of-the-art technique on all aspects, including precision, recall, efficiency, space overhead and flexibility of accommodating different semantics.

Multi-dimensional data indexing has received much attention in a centralized database. However, not so much work has been done on this topic in the context of Peerto- Peer systems. In this paper, we propose a new Peer-to- Peer framework based on a balanced tree structure overlay, which can support extensible centralized mapping methods and query processing based on a variety of multidimensional tree structures, including R-Tree, X-Tree, SSTree, and M-Tree. Specifically, in a network with N nodes, our framework guarantees that point queries and range queries can be answered within O(logN) hops. We also provide an effective load balancing strategy to allow nodes to balance their work load efficiently. An experimental assessment validates the practicality of our proposal.

Peer-to-Peer systems have recently become a popular means to share resources. Effective search is a critical requirement in such systems, and a number of distributed search structures have been proposed in the literature. Most of these structures provide "log time search" capability, where the logarithm is taken base 2. That is, in a system with N nodes, the cost of the search is O(log2N).In database systems, the importance of large fanout index structures has been well recognized. In P2P search too, the cost could be reduced considerably if this logarithm were taken to a larger base. In this paper, we propose a multi-way tree search structure, which reduces the cost of search to O(logmN), where m is the fanout. The penalty paid is a larger update cost, but we show how to keep this penalty to be no worse than linear in m. We experimentally explore this tradeoff between search and update cost as a function of m, and suggest how to find a good trade-off point.The multi-way tree structure we propose, BATON*, is derived from the BATON structure that has recently been suggested. In addition to multi-way fanout, BATON* also adds support for multi-attribute queries to BATON.