Reproducible benchmarking of tools that automatically detect vulnerabilities in source code remains challenging due to inconsistent implementations, varying data preprocessing, and methodological flaws that compromise fair model comparison. In a recent study, 9 in 10 vulnerability detection studies were found to use inappropriate evaluation approaches, with models achieving high scores through spurious correlations rather than actual vulnerability detection. We present VulnBench, an extensible, open-source benchmarking tool that enables fair comparison across models and datasets. Our systematic evaluation of CodeBERT, GraphCodeBERT, CodeT5 (encoder-only and full), and NatGen across eight mostly C/C++ source code datasets reveals that proper threshold optimization can improve F1-scores by up to 54%, as well as wide variation in F1-scores showing the large gap in the difficulty of the vulnerability dataset field. By standardising evaluation protocols, VulnBench enables researchers to distinguish between genuine model improvements and methodological artifacts as well as reducing wasteful duplication of effort spent on reproducing results.

Score-at-a-Time (SaaT) retrieval, utilising impact-ordered indexes, remains a relatively overlooked search strategy with increasing importance. Within the context of learned sparse representations and approximate retrieval, SaaT has proven to be a competitive alternative to the popular Document-at-a-Time (DaaT) approach. Yet, there are only two notable implementations of SaaT search engines: JASSv2 and IOQP. We are interested in the differences between these systems and how those differences affect performance. We identify differences in postings lists due to indexing, ranking functions, and quantization. Thus, we introduce ciffTools for quantizing the ciff indexes used by both search engines; eliminating these differences. Then, in a reproducibility study we reproduce previous experiments investigating the efficiency gap between the systems. They also differ in their compression codecs, with IOQP using SIMD BP-128 and JASSv2 using Elias Gamma SIMD VB. We, unexpectedly, find in another reproducibility experiment that SIMD BP-128 and QMX outperform Elias Gamma SIMD VB in situ. Finally, we investigate the CPU effect, and find the engines are affected differently. Overall, JASSv2 has faster median latency on a server-grade CPU, while on a desktop-grade they are evenly matched. IOQP has faster tail latency regardless of CPU. Our work reduces the throughput difference between JASSv2 and IOQP and offers insights into which aspects affect the efficiency of SaaT.

Electroencephalogram (EEG)-based Brain-Computer Interfaces (BCIs) build a communication path between human brain and external devices. Among EEG-based BCI paradigms, the most commonly used one is motor imagery (MI). As a hot research topic, MI EEG-based BCI has largely contributed to medical fields and smart home industry. However, because of the low signal-to-noise ratio (SNR) and the non-stationary characteristic of EEG data, it is difficult to correctly classify different types of MI-EEG signals. Recently, the advances in Deep Learning (DL) significantly facilitate the development of MI EEG-based BCIs. In this paper, we provide a systematic survey of DL-based MI-EEG classification methods. Specifically, we first comprehensively discuss several important aspects of DL-based MI-EEG classification, covering input formulations, network architectures, public datasets, etc. Then, we summarize problems in model performance comparison and give guidelines to future studies for fair performance comparison. Next, we fairly evaluate the representative DL-based models using source code released by the authors and meticulously analyse the evaluation results. By performing ablation study on the network architecture, we found that (1) effective feature fusion is indispensable for multi-stream CNN-based models. (2) LSTM should be combined with spatial feature extraction techniques to obtain good classification performance. (3) the use of dropout contributes little to improving the model performance, and that (4) adding fully connected layers to the models significantly increases their parameters but it might not improve their performance. Finally, we raise several open issues in MI-EEG classification and provide possible future research directions.

Depression has long been described as a common mental health disorder and a disease with a set of diagnostic criteria that influences the affected individuals' feelings and behavior. The prevalence of Internet use has augmented people’s openness to share their experiences and struggles, including mental health disorders on social media thus researchers have tried developing classification models for depression detection using various machine learning and deep learning techniques. In this research, we propose a deep learning architecture with an attention mechanism on CNN-BiLSTM (CBA) and provide a comparative analysis to benchmark well-known deep learning models using the public dataset namely CLEF2017. We found that along with F1 score, precision and recall it is also vital to consider the Area under the curve - Receiver operating characteristic curve (AUC-ROC) and Mathews Correlation Coefficient (MCC) metrics for evaluating depression classification models since the MCC considers all the four values of a confusion matrix. Based on our experiments, the CBA model outperforms the existing state of the art model with an overall accuracy of 96.71% and scores of 0.85 and 0.77 for AUC-ROC and MCC, respectively.

There is a lack of reproducibility in results from experiments that apply the Appraisal taxonomy. Appraisal is widely used by linguists to study how people judge things or people. Automating Appraisal could be beneficial for use cases such as moderating online comments. Past work in Appraisal annotation has been descriptive in nature and, the lack of publicly available data sets hinders the progress of automation. In this work, we are interested in two things; first, measuring the performance of automated approaches to Appraisal classification in the publicly available Australasian Language Technology Association (ALTA) Shared Task Challenge data set. Second, we are interested in reproducing the annotation of the ALTA data set. Four additional annotators, each with a different linguistics background, were employed to re-annotate the data set. Our results show a poor level of agreement at more detailed Appraisal categories (Fleiss Kappa = 0.059) and a fair level of agreement (Kappa = 0.372) at coarse-level categories. We find similar results when using automated approaches that are available publicly. Our empirical evidence suggests that at present, automating classification is practical only when considering coarse-level categories of the taxonomy.

In recent years, various approaches for multiresident human activity recognition (HAR) in a smart indoor environment have been developed and improved along with the rapid development of sensors and AI technologies. Research in data stream-based online learning (OL) for multiresident HAR is relatively new and a majority of the existing works have been developed based on training batches of data that cannot recognize real-time activities. To address the challenges of OL for multiresident HAR, we propose a novel OL architecture based on a locally weighted ensemble detection-based adaptive random forest (LED-ARF) classifier. We conduct a comprehensive performance comparison of eight famous OL classification techniques and our LED-ARF method. The comparison is evaluated based on the two benchmarking CASAS and ARAS data sets. Our experimental results show that LED-ARF achieves the best performance with the highest robustness for online multiresident HAR.

The common approach for introducing middle school and high school students to computational thinking and engineering is through programming or robotics activities that were designed without much social context. Therefore, one of the challenges such outreach programs pose is further stereotyping of students who are interested in computer science, by presenting them as people who ought to be hyper-intelligent and very much focused on computers with no social skills. Realizing the above, and wanting to attract a diverse audience to study computer science, we developed a one-day workshop which showcases the interdisciplinary nature of computer science and the various ways computing can help students discover and solve societal problems. We chose multiculturalism, cyber security and air pollution as the context of our computational thinking activities. This paper describes our experiences in developing and teaching the workshops that runs three to five times a year, hosted either in our institute or at local schools. Regardless of students' previous STEM experiences and socioeconomic statuses, both students and teachers reported that the knowledge gained enabled them to either create a computer program or a robot and also led to a better understanding of how computers can be used in daily life. In other words, the workshop successfully opened the minds and perspectives of the middle school and high school students on computer science. Additionally, over time, we also observed an increased number of students and schools participating in such workshops.

Behavior and activities awareness of people within a smart house is studied using pressure sensors on the floor. In turn, Machine Learning algorithms are used to predict unusual behavior of the house residents. The research focuses in people with supported needs trying to live a normal life in their own houses, allowing them more independence. The system relies on using pressure sensors on the floor to estimate the position of a person in some regions of interest, by measuring their presence on the sensors. System is designed as an Internet of Things (IOT) platform. An array of sensors on the floor are connected through routing devices to an Internet gateway that publishes the data to a Cloud System and into a Database. Machine learning (ML) is used to predict the activities of a person (standing, sitting, lying down), and to determine whether their behavior is common or not. IOT enables technologies that allow having remote monitoring of activities and behavior of residents. Unlike other researches done using pressure sensors that concentrates on predicting people's activities on small area of the room, we are utilizing the sensors to predict people's activities and behavior on all areas of the room.

We have developed a smart home system that estimates people's positions, activities and behavior using pressure sensors. The main difference between our research and state-of-the-art in the field is the use of a reduced number of pressure sensors to allow for a larger sensing area. An Internet of Things (IOT) platform in hand with Machine Learning (ML) algorithms are used to route signals and process them in the Cloud to predict activities and behavior and infer whether they are usual or not.

There have been many proposed smart home systems that utilize ultrasonic, pressure or passive infrared sensors to predict a resident's activities in a single occupant house, such as sitting, eating, cooking, walking and lying down. However, these systems require extensive training and manual labeling of the resident's activities before they can be used in the resident's house-making the systems inflexible and expensive. This paper proposes a more realistic system that relies on machine learning techniques to identify a resident's activities based on the ultrasonic sensors' readings and on/off state without the need to manually label the resident's prior activities. The system is trained on a single resident, then tested with the same occupant and with different people. Our evaluation shows that not only can our system accurately predict a resident's activities when it is trained on the resident's previous activities, but our system can also accurately predict the resident's activities even when it is trained on someone else's activities. This research enables this smart home system to be widely adopted in people's houses with minimal training.

When developing smart home system techniques for predicting a person's activities most are trained and evaluated by the same group of people. However, in practice, it is difficult and expensive to manually label each activity the participants perform. In this poster, we describe the findings of our initial investigations on simple features that we can use to predict an inhabitant's activities based on the machine learning model that is trained on activities done by different residents. Our study indicates that ultrasonic sensors' readings, duration and on/off states can accurately predict (86%) of the other resident's activities, even when the activities are interleaved with each other.