Efficient big data management is a dire necessity to manage the exponential growth in data generated by digital information systems to produce usable knowledge. Structured databases, data lakes, and warehouses have each provided a solution with varying degrees of success. However, a new and superior solution, the data Lakehouse, has emerged to extract actionable insights from unstructured data ingested from distributed sources. By combining the strengths of data warehouses and data lakes, the data Lakehouse can process and merge data quickly while ingesting and storing high-speed unstructured data with post-storage transformation and analytics capabilities. The Lakehouse architecture offers the necessary features for optimal functionality and has gained significant attention in the big data management research community. In this paper, we compare data lake, warehouse, and lakehouse systems, highlight their strengths and shortcomings, identify the desired features to handle the evolving challenges in big data management and analysis and propose an advanced data Lakehouse architecture. We also demonstrate the performance of three state-of-the-art data management systems namely HDFS data lake, Hive data warehouse, and Delta lakehouse in managing data for analytical query responses through an experimental study.

Digital information systems currently generate a vast amount of data every minute which emphasizes the continuing need to advance big data management systems with efficient data ingestion and knowledge extraction capabilities. To address the ‘big data’ problems due to high volume, velocity, variety, and veracity, data management systems evolved from structured databases to big data storage systems, graph databases, data warehouses, and data lakes but each solution has its strengths and shortcomings. The need to produce actionable knowledge fast from unstructured data ingested from distributed sources requires a marriage of data warehouses and data lakes to create a data Lakehouse (LH). The objective is to use the strengths of the data warehouse in producing insights fast from processed merged data, and of the data lake in ingesting and storing high-speed unstructured data with post-storage transformation and analytics capabilities. In this paper, we present a comparative review of the existing data warehouse and data lake technology to highlight their strengths and weaknesses and propose the desired and necessary features of the LH architecture, which has recently gained a lot of attention in the big data management research community.

During recent years, Question Answering (QA) systems have been widely used in many industries to provide round the clock online services to consumers from all over the world. The importance of such services became more evident during the pandemic in online medical services, education, training, marketing, system support and administration. Most of the existing systems apply simple rule-based QA strategy. Human-defined rules are used to apply pattern matching for extracting information from a given data or knowledge base to generate responses to user queries. However, rule-based pattern matching techniques are not intelligent enough to understand the context of the question to always generate appropriate responses and are static. In this work, we explored different data preprocessing strategies and BERT-style pre-trained models to build an information retrieval (IR)-based Domain specific QA framework named BIRD-QA, and created a domain specific knowledge base using website data of a university department. We implemented multiple variations of extended BERT and ALBERT-base models and validated our framework on reading comprehension task using the Stanford Question Answering Dataset (SQuAD) 1.1 and 2.0 datasets. Our extended ALBERT-based model achieved 75.4% Exact Match (EM) score and 78.8% F1 score. We also present a small feasibility test of our framework for departmental QA using data from a university website.

Drivable area or free space detection is an important task in Advanced Driver-Assistance Systems (ADAS) and autonomous driving system. It can help intelligent vehicles understand road conditions and determine safe driving area. Semantic segmentation is a pixel-wise prediction which can classify each pixel into its category. In this paper, we propose a deep learning-based semantic segmentation architecture to predict the drivable area in front of the vehicle. Our model is built based on ResNet backbone with the Feature Pyramid Network (FPN) and Atrous Spatial Pyramid Pooling (ASPP) modules. The backbone in the bottom-up architecture extracts features and an ASPP is attached to the last decoder layer. Additionally, a top-down architecture with lateral connections is added in the decoder and the FPN utilizes the multi-scale features for final prediction. Our model is evaluated on the Cityscapes street scene dataset and achieves 95.90% mIoU on road segmentation. Next, the model is evaluated on the BDD100K large-scale diverse driving dataset with direct drivable region and alternative drivable region annotations. For this dataset our model achieves 84.58% mIoU which is comparable to some State-of-the-Art models.

Vital signs are important parameters that can reflect people's physiological status and help physicians provide medical advice. Remote Photoplethysmography (rPPG) is a fast, low-cost and convenient method to remotely collect biometric data, and requires only a facial video recorded using a smartphone or other camera. Remote medical service provisioning proved to be a dire need during the COVID-19 pandemic. To leverage the cloud-based medical advice provisioning platform of Your Doctors Online, we propose a rPPG methodology to measure people's Heart Rate (HR) and Heart Rate Variability (HRV) based on a facial video recorded by the users using a smartphone. We validate our model on the TokyoTech remote PPG dataset.

An essential part of building a data-driven organization is the ability to handle and process continuous streams of data to discover actionable insights. The explosive growth of interconnected devices and the social Web has led to a large volume of data being generated on a continuous basis. Streaming data sources such as stock quotes, credit card transactions, trending news, traffic conditions, time-sensitive patient's data are not only very common but can rapidly depreciate if not processed quickly. The ever-increasing volume and highly irregular nature of data rates pose new challenges to data stream processing systems. One such challenging but important task is how to accurately ingest and integrate data streams from various sources and locations into an analytics platform. These challenges demand new strategies and systems that can offer the desired degree of scalability and robustness in handling failures. This paper investigates the fundamental requirements and the state of the art of existing data stream ingestion systems, propose a scalable and fault-tolerant data stream ingestion and integration framework that can serve as a reusable component across many feeds of structured and unstructured input data in a given platform, and demonstrate the utility of the framework in a real-world data stream processing case study that integrates Apache NiFi and Kafka for processing high velocity news articles from across the globe. The study also identifies best practices and gaps for future research in developing large-scale data stream processing infrastructure.

Prediction of stock markets is a challenging problem because of the number of potential variables as well as unpredictable noise that may contribute to the resultant prices. However, the ability to analyze stock market trends could be invaluable to investors and researchers, and thus has been of continued interest. Numerous statistical and machine learning techniques have been explored for stock analysis and prediction. We present a comparative study of two very promising artificial neural network models namely a Long Short-Term Memory (LSTM) recurrent neural network (RNN) and a deep neural network (DNN) in forecasting the daily and weekly movements of the Indian BSE Sensex index. With both networks, measures were taken to reduce overfitting. Daily predictions of the Tech Mahindra (NSE: TECHM) stock price were made to test the generalizability of the models. Both networks performed well at making daily predictions, and both generalized well to make daily predictions of the Tech Mahindra data. The LSTM RNN outperformed the DNN in terms of weekly predictions and thus, holds more promise for making longer term predictions.

Stock market forecasting is very important in the planning of business activities. Stock price prediction has attracted many researchers in multiple disciplines including computer science, statistics, economics, finance, and operations research. Recent studies have shown that the vast amount of online information in the public domain such as Wikipedia usage pattern, news stories from the mainstream media, and social media discussions can have an observable effect on investors' opinions towards financial markets. The reliability of the computational models on stock market prediction is important as it is very sensitive to the economy and can directly lead to financial loss. In this paper, we retrieved, extracted, and analyzed the effects of news sentiments on the stock market. Our main contributions include the development of a sentiment analysis dictionary for the financial sector, the development of a dictionary-based sentiment analysis model, and the evaluation of the model for gauging the effects of news sentiments on stocks for the pharmaceutical market. Using only news sentiments, we achieved a directional accuracy of 70.59% in predicting the trends in short-term stock price movement.

Graph processing is one of the most important topics in big data processing. The graph architecture is suitable for distributed processing as the processing works in an iterative manner allowing parallelism. Also, the structure has proved to be suitable in representing social networks, web page indexes, and many other problems. However, graph processing introduce many problems as well. Partitioning the graph to distribute the data on multiple machines and minimizing data movement is a serious challenge. Also many of the graph algorithms have high complexity. GraphX is one of the frameworks that introduce an abstraction on top of Spark, an iterative data processing engine. However, GraphX and other novel graph abstractions still do not support processing data streams with online graphs. In this work we try to use IndexedRDD, a library to enable fine grained updates as a key-value store on top of Spark to represent a graph structure and test if it can be used as an efficient online graph storage for spark streaming. We did experiments to compare our data streaming implementation using IndexedRDD with the obvious elementary solution of using RDD transformations to join the old RDD with the new one to make a new composite RDD on each micro-batch. We also want to compare the above two with a distributed in-memory key-value store (such as Redis). The results show big advantage of using Redis over RDD transformations and IndexedRDD. However, it has some limitations such as lacking the support for property graphs. IndexedRDD, on the other hand, has shown good performance for insertions and a shortcoming in its need to rebuild the index after each data update, which add extra time on each lookup that cannot be tolerated when lookup speed is essential.

Cloud-based Analytics-as-a-Service (CLAaaS) was developed by Zulkernine et al. with a goal to simplifying big data analytics users. It provides software-as-a-service access to a variety of back end analytics tools and data stores. One of the tools is the Workflow Instance Generation and Selection (WINGS). WINGS allows users to reuse predefined workflows and their components containing semantic meta-data to define new workflows; late binding of the workflows to data at the time of execution to enable the use of most recent data, and definition of domain specific software code as custom analytic components in workflows. How ever, the data used in WINGS for the workflows are mostly flat files that are stored on the WINGS server or shared directories. The goal of this project is to add support for big data storage systems to WINGS and validate the extensions using multiple data analytic workflows of different complexities with data residing in a variety of back end data sources. The extension allows the CLAaaS users to create, validate and execute analytic workflows in a distributed environment and use data from multiple big data storage systems. We validate our work using four big data storage systems in WINGS workflows namely, Apache HBase, MongoDB, MySQL with a front-end interface.

Predicting stock prices using computer generated models has been a popular research topic and has also been widely explored. However, the connectivity of the global financial market, availability of big data in multiple domains that influence the financial market, accessibility of information in real time and the demand for fast analytics continue to offer new research challenges. One of the complexities stems from the numerous ways in which we seek to set prediction parameters, whether it is the difference in an individual stocks' growth pattern or the time frame in which the predictions occur. The level of complexity has created a trend towards more advanced techniques in this field namely, the research into developing hybrid models that are composed of multiple prediction models with a view to yield more accurate results. The Proposed Hybrid Model (PHM) used in this paper is a combination of an Exponential Smoothing Model (ESM), an Auto Regressive Integrated Moving Average (ARIMA) model, and a Back-propagation Neural Network (BPNN) model. PHM combines the predictions of each of the component model based on weights assigned by a genetic algorithm, which is designed to provide an optimum output. In this paper, we seek to use the S&P 400 and 500 indexes to train and test the PHM to find daily closing values. For comparison of the results, Directional Accuracy (DA) is used as a metric. It was found that the results for the ARIMA and ESM on daily stock index data were far less accurate than that of the BPNN, which received comparable results to the baseline. However, due to the poor results of the ARIMA and ESM the hybrid model showed no significant results for the data and was inferior to the baseline.

Enormous amounts of data are generated everyday of both structured and unstructured nature. Regardless of their differences, data sources must be used in tandem in any effective big data operation. This paper proposes a Software as a Service (SaaS) framework called BINARY which provides a back-end infrastructure for ad-hoc querying, accessing, visualizing and joining data from different data sources such as Relational Database Management Systems like MySQL and big data storage systems like Apache Hive. BINARY is extendable and allows adding other storage engines (e.g. HBase) and analytics engines (e.g. R) as needed. A REST software architecture is used in the framework to enable loose connections between the engines and user interface programs to facilitate their independent updates without affecting the data infrastructure. Our approach is validated with a proof-of-concept prototype implemented on the OpenStack cloud system.