Generative large language models (such as ChatGPT) are increasingly influencing various aspects of our lives, partly due to their training on vast datasets that encompassing big data paradigms and range of topics. "Intervista," an award-winning Italian film by Federico Fellini, focuses on his interview with a Japanese TV crew. Inspired by this, we conducted interviews with a diverse set of open-source and OpenAI models to explore various political, economic, and cultural aspects of life, evaluating LLM performance. We also examined whether a correlation exists between a country’s GDP per capita and the quality of the model’s answers. To this end, we utilized a Huggingface model leaderboard to select appropriate models and deployed them in an AWS SageMaker GPU environment. The identical questions were posed about nearly 200 countries, and the responses were analyzed to verify their accuracy and correlation with Gross Domestic Product (GDP). We were amazed by the diversity, quantity, and quality of existing pretrained open-source LLMs. Our journey provided insights into model selection, inference pipeline automation, GPU configuration, generated texts benchmarking, and systematic evaluation of model quality. Overall, leading LLMs performed well, providing reasonable responses for many countries. However, we discovered that the depth and detail of the answers were influenced by a country’s GDP per capita, with higher-income nations receiving more accurate responses.

Novel Coronavirus (COVID-19) has changed the life of the planet. It is extremely important to monitor the situation in real time. Our secure methodology can help people to trace the situation in their country or state without touching a single computer key, just using voice-first computing devices. We have been using the data from multiple sources, created a suite of Alexa skills (Voice-first applications) and observed how the data evolves. The coronavirus data is downloaded automatically to the AWS Cloud and stored securely in the No-SQL DynamoDB database and S3 buckets to help users to monitor up-to-date statistics. Moreover, Alexa Echo devices with screens will display comprehensive graphs containing the most vital numbers – new cases, new deaths, mortality rate and hospitalizations since the pandemic started. Our system is safe, secure, automatic and resilient. It helps users to maintain social distancing and obtain up-to-date information about coronavirus in the location of interest without a single touch, just by using voice. During our journey we have designed and implemented many convenient commands, improving usability and multi-modal user experience. Our innovative approach, serverless architecture and Big Data methodology can help millions of people to stay on top of the coronavirus situation and make day-to-day choices using the information provided. It can also help officials to make educated decisions about opening certain businesses, institutions or activities. Since more and more voice assistants (AI devices) appear in public places – hotels, restaurants, and airports, our approach will help people to stay informed everywhere. Using our touch-free Alexa analytical skills will also promote social distancing.

Deep Neural Networks are used to solve the most challenging world problems. In spite of the numerous advancements in the field, most of the models are being tuned manually. Experienced Data Scientists have to manually optimize hyperparameters, such as dropout rate, learning rate or number of neurons for Big Data applications. We have implemented a flexible automatic real-time hyperparameter tuning methodology. It works for arbitrary models written in Python and Keras. We also utilized state of the art Cloud services such as trigger based serverless computing (Lambda), and advanced GPU instances to implement automation, reliability and scalability.The existing tuning libraries, such as hyperopt, Scikit-Optimize or SageMaker, require developers to provide a list of hyperparameters and the range of their values manually. Our novel approach detects potential hyperparameters automatically from the source code, updates the original model to tune the parameters, runs the evaluation in the Cloud on spot instances, finds the optimal hyperparameters, and saves the results in the No-SQL database. The methodology can be applied to numerous Big Data Machine Learning systems.

While adopting Blockchain technologies to automate their enterprise functionality, organizations are recognizing the challenges of scalability and manual configuration that the state of art present. Scalability of Hyperledger Fabric is an open challenge recognized by the research community. We have automated many of the configuration steps of installing Hyperledger Fabric Blockchain on AWS infrastructure and have benchmarked the scalability of that system. We have used the UCR (University of California Riverside) Time Series Archive with 128 timeseries datasets containing over 191,177 rows of data totaling 76,453,742 numbers. Using an automated Serverless approach, we have loaded this dataset, by chunks, into different AWS instances, triggering the load by SQS messaging. In this paper, we present the results of this benchmarking study and describe the approach we took to automate the Hyperledger Fabric processes using serverless Lambda functions and SQS triggering. We will also discuss what is needed to make the Blockchain technology more robust and scalable.

Saving nature using Big Data Analytics is a very noble goal. Using New York taxi rides data, we decided to learn how many rides could be consolidated. It was a journey we would like to share. First, we had to choose the platform for calculation between Amazon Athena, Serverless Microservices, SQL or NoSql databases, Hadoop and Spark. Then, we had to find an optimal solution for the platform using assorted tuning and optimization techniques. Although the problem seems to be straight forward, it turned out that the solution is quite challenging because of the input size, data quality, calculation complexities and numerous EMR/Spark tuning options. We have been using New York taxi data from 2009 to 2017 to quantify the rides that can be joined together. The taxi rides were consolidated based on pickup location, pickup time and drop-off location. We have been calculating the percentage of taxi rides that can be joined. The benchmark originally set was rides within five minutes with a pickup and drop-off locations within half a kilometer. Then we started experimenting with different times and locations. We have been using parquet format, parallel Scala collections, compression, filtering, new column introduction, tuning parameters, I/O overhead tuning, bucketing, timeouts and partitioning. Over 1.2 billion rides were processed using Amazon EMR with Spark. We have been optimizing calculation time and processing price. Spark has hundreds of parameters, EMR has over fifty instances to choose from. It was challenging to process our data within reasonable time. We were able to find the optimal Spark queries (plans), tested different types of joins and compared their performances. Also, we were able to compare I/O and in-memory operations during partitioning and large files manipulation (the input file sizes were hundreds of Gigabytes). The results were amazing - we could consolidate around thirty five percent of total rides, saving tons of gas and improving environment and traffic in New York City.

Machine Learning algorithms on large datasets can be executed in the Cloud. Amazon Web Services (AWS) provides over 60 different On-Demand EC2 instances [1]. The instance prices range from $0.0059 (t2.nano) to $14.4 (p2.16xlarge) per hour. We decided to build an automatic recommendation system to choose the best instance for a dataset and a machine learning algorithm to optimize time and money spent. After running multiple algorithms for different Big Data sets on assorted AWS instances and collecting the results in the NoSQL DynamoDB database, we have trained machine learning models to predict time and cost using assorted regression ML methods.