Generating realistic and controllable vehicle trajectories remains a key challenge in traffic simulation due to the dynamic and stochastic nature of road traffic. Existing deep learning approaches often rely solely on raw sensory inputs, limiting their ability to capture both fine-grained spatiotemporal dependencies and highlevel semantic context. To address this gap, we propose a generalizable framework with two complementary modules. The first performs low-level numerical modeling using a dynamic Spatial-Temporal Graph Neural Network and visual encoder to capture spatial-temporal dynamics and environmental context. The second applies perception-driven semantic modeling via a Multi-Modal Large Language Model pipeline to extract human-like interpretations of traffic scenarios. Outputs from both modules condition a diffusion-based generative model to produce behaviorally feasible, controllable trajectories. Experiments on a real-world dataset show significant improvements over state-of-the-art baselines across multiple metrics, validating the effectiveness of our approach.

In the current social media landscape, the study of influence propagation and consensus formation has gained prominence. While user interactions like retweeting are apparent, the underlying pathways of influence often remain hidden and complex. This study proposes a novel network called Latent Influence Network (LIN), which advances the analysis of influence on social media. LIN's architecture and the process of parameter selection are meticulously discussed within the comprehensive Latent Influence Detection Framework (LIDET). Based on the user's behavior label, LIN identifies the optimal network configuration, revealing more accurate influence patterns. We applied the LIDET framework to four diverse datasets, each demonstrating substantial improvements in influence pattern recognition over traditional network models. Specifically, in a case study on a COVID-19 dataset, LIN achieved a classification accuracy of 99%, significantly outperforming conventional methods. These findings underscore the utility of LIN in capturing the dynamics of influence and enhancing our understanding of opinion formation on social media.

The rise of social media has been accompanied by a dark side with the ease of creating fake accounts and disseminating misinformation through coordinated attacks. Existing methods to identify such attacks often rely on thematic similarities or network-based approaches, overlooking the intricate causal relationships that underlie coordinated actions. This work introduces a novel approach for detecting coordinated attacks using Convergent Cross Mapping (CCM), a technique that infers causality from temporal relationships between user activity. We build on the theoretical framework of CCM by incorporating topic modelling as a basis for further optimizing its performance. We apply CCM to real-world data from the infamous IRA attack on US elections, achieving F1 scores up to 75.3% in identifying coordinated accounts. Furthermore, we analyse the output of our model to identify the most influential users in a community and uncover leader-follower dynamics based on inferred causal relationships. We also demonstrate how our method reveals coordinated behaviour across different time periods, including campaigns predating the 2016 elections. We apply our model to a case study involving COVID-19 anti-vax related discussions on Twitter. Our results demonstrate the effectiveness of our model in uncovering causal structures of coordinated behaviour, offering a promising avenue for mitigating the threat of malicious campaigns on social media platforms.

The increasing availability of urban data offers new opportunities for learning region representations, which can be used as input to machine learning models for downstream tasks such as check-in or crime prediction. While existing solutions have produced promising results, an issue is their fixed formation of regions and fixed input region features, which may not suit the needs of different downstream tasks. To address this limitation, we propose a model named FlexiReg for urban region representation learning that is flexible with both the formation of urban regions and the input region features. FlexiReg is based on a spatial grid partitioning over the spatial area of interest. It learns representations for the grid cells, leveraging publicly accessible data, including POI, land use, satellite imagery, and street view imagery. We propose adaptive aggregation to fuse the cell representations and prompt learning techniques to tailor the representations towards different tasks, addressing the needs of varying formations of urban regions and downstream tasks. Extensive experiments on five real-world datasets demonstrate that FlexiReg outperforms state-of-the-art models by up to 202% in term of the accuracy of four diverse downstream tasks using the produced urban region representations.

During times of crisis, social media platforms play a crucial role in facilitating communication and coordinating resources. In the midst of chaos and uncertainty, communities often rely on these platforms to share urgent pleas for help, extend support, and organize relief efforts. However, the overwhelming volume of conversations during such periods can escalate to unprecedented levels, necessitating the automated identification and matching of requests and offers to streamline relief operations. Additionally, there is a notable absence of studies conducted in multilingual settings despite the fact that any geographical area can have a diverse linguistic population. Therefore, we proposecrisisresponsematcher (CReMa), a systematic approach that integrates textual, temporal, and spatial features to address the challenges of effectively identifying and matching requests and offers on social media platforms during emergencies. Our approach utilizes a crisis-specific pretrained model and a multilingual embedding space. We emulate human decision-making to compute temporal and spatial features and nonlinearly weigh the textual features. The results from our experiments are promising, outperforming strong baselines. Additionally, we introduce a novel multilingual dataset, simulating help-seeking and offering assistance on social media in 16 languages, and conduct comprehensive cross-lingual experiments. Furthermore, we analyze a million-scale geotagged global dataset to understand patterns in seeking help and offering assistance on social media. Overall, these contributions advance the field of crisis informatics and provide benchmarks for future research in the area.

Serverless computing has gained popularity as a novel cloud execution model for applications in recent times. Businesses constantly try to leverage this new paradigm to add value to their revenue streams. The serverless eco-system accommodates many application domains successfully. However, its inherent properties such as cold start delays and relatively high per unit charges appear as a shortcoming for certain application workloads, when compared to a traditional Virtual Machine (VM) based execution scenario. A few research works exist, that study how serverless computing could be used to mitigate the challenges in a VM based cluster environment, for certain applications. In contrast, this work proposes a generalized framework for determining which workloads are best able to reap benefits of a serverless computing environment. In essence, we present a potential hybrid scheduling solution for exploiting the benefits of both a serverless and a VM based serverful computing environment. Our proposed framework leverages the actor-critic based deep reinforcement learning architecture coupled with the proximal policy optimization technique, in determining the best scheduling decision for workload executions. Extensive experiments conducted demonstrate the effectiveness of such a solution, in terms of user cost and application performance, with improvements of up to 44% and 11% respectively.

An increasing number of related urban data sources have brought forth novel opportunities for learning urban region representations, i.e., embeddings. The embeddings describe latent features of urban regions and enable discovering similar regions for urban planning applications. Existing methods learn an embedding for a region using every different type of region feature data, and subsequently fuse all learned embeddings of a region to generate a unified region embedding. However, these studies often overlook the significance of the fusion process. The typical fusion methods rely on simple aggregation, such as summation and concatenation, thereby disregarding correlations within the fused region embeddings. To address this limitation, we propose a novel model named HAFusion. Our model is powered by a dual-feature attentive fusion module named DAFusion, which fuses embeddings from different region features to learn higher-order correlations be-tween the regions as well as between the different types of region features. DAFusion is generic - it can be integrated into existing models to enhance their fusion process. Further, motivated by the effective fusion capability of an attentive module, we propose a hybrid attentive feature learning module named HALearning to enhance the embedding learning from each individual type of region features. Extensive experiments on three real-world datasets demonstrate that our model HAFusion outperforms state-of-the-art models across three different prediction tasks. Using our learned region embeddings leads to consistent and up to 31 % improvements in the prediction accuracy.

The proliferation of misinformation and disinformation on social media networks has become increasingly concerning. With a significant portion of the population using social media on a regular basis, there are growing efforts by malicious organizations to manipulate public opinion through coordinated campaigns. Current methods for identifying coordinated user accounts typically rely on either similarities in user behaviour, latent coordination in activity traces, or classification techniques. In our study, we propose a framework based on the hypothesis that coordinated users will demonstrate abnormal growth in their behavioural patterns over time relative to the wider population. Specifically, we utilize the EPClose algorithm to extract contrasting patterns of user behaviour during a time window of malicious activity, which we then compare to a historical time window. We evaluated the effectiveness of our approach using real-world data, and our results show a minimum increase of 10% in the F1 score compared to existing approaches.

Social media platforms play an essential role in crisis communication, but analyzing crisis-related social media texts is challenging due to their informal nature. Transformer-based pre-trained models like BERT and RoBERTa have shown success in various NLP tasks, but they are not tailored for crisis-related texts. Furthermore, general-purpose sentence encoders are used to generate sentence embeddings, regardless of the textual complexities in crisis-related texts. Advances in applications like text classification, semantic search, and clustering contribute to the effective processing of crisis-related texts, which is essential for emergency responders to gain a comprehensive view of a crisis event, whether historical or real-time. To address these gaps in crisis informatics literature, this study introduces CrisisTransformers, an ensemble of pre-trained language models and sentence encoders trained on an extensive corpus of over 15 billion word tokens from tweets associated with more than 30 crisis events, including disease outbreaks, natural disasters, conflicts, and other critical incidents. We evaluate existing models and CrisisTransformers on 18 crisis-specific public datasets. Our pre-trained models outperform strong baselines across all datasets in classification tasks, and our best-performing sentence encoder improves the state-of-the-art by 17.43% in sentence encoding tasks. Additionally, we investigate the impact of model initialization on convergence and evaluate the significance of domain-specific models in generating semantically meaningful sentence embeddings. The models are publicly available at: https://huggingface.co/crisistransformers.

The emergence of social media as one of the main platforms for people to access news has enabled the wide dissemination of fake news, having serious impacts on society. Thus, it is really important to identify fake news with high confidence in a timely manner, which is not feasible using manual analysis. This has motivated numerous studies on automating fake news detection. Most of these approaches are supervised, which requires extensive time and labour to build a labelled dataset. Although there have been limited attempts at unsupervised fake news detection, their performance suffers due to not exploiting the knowledge from various modalities related to news records and due to the presence of various latent biases in the existing news datasets (e.g., unrealistic real and fake news distributions). To address these limitations, this work proposes an effective framework for unsupervised fake news detection, which first embeds the knowledge available in four modalities (i.e., source credibility, textual content, propagation speed, and user credibility) in news records and then proposes$(UMD)^{2}$, a novel noise-robust self-supervised learning technique, to identify the veracity of news records from the multi-modal embeddings. Also, we propose a novel technique to construct news datasets minimizing the latent biases in existing news datasets. Following the proposed approach for dataset construction, we produce a Large-scale Unlabelled News Dataset consisting 419,351 news articles related to COVID-19, acronymed asLUND-COVID. We trained the proposed unsupervised framework usingLUND-COVIDto exploit the potential of large datasets, and evaluate it using a set of existing labelled datasets. Our results show that the proposed unsupervised framework largely outperforms existing unsupervised baselines for different tasks such as multi-modal fake news detection, fake news early detection and few-shot fake news detection, while yielding notable improvements for unseen domains during training.

Abstract With the increasing connectedness of vehicles, real-time spatio-temporal data can be collected from citywide road networks. Innovative data management solutions can process the collected data for the purpose of reducing travel time. However, a majority of the existing solutions have missed the opportunity to better manage the collected data for improving road safety at the network level. We propose an efficient data management framework that uses network-level data to improve road safety for citywide applications. Our framework uses a graph-based data structure to maintain real-time network-level traffic data. Based on the graph, the framework uses a novel technique to generate driving instructions for individual vehicles. By following the instructions, inter-vehicular spacing can be increased, leading to an improvement of road safety. Experimental results show that our framework improves road safety, measured based on the time to collision between vehicles, from the state-of-the-art traffic data management solutions by a large margin while achieving lower travel times compared with the solutions. The framework is also readily deployable for large-scale real-time applications due to its low computation costs.

Dynamic road network optimization has been used for improving traffic flow in an infrequent and localized manner. The development of intelligent systems and technology provides an opportunity to improve the frequency and scale of dynamic road network optimization. However, such improvements are hindered by the high computational complexity of the existing algorithms that generate the optimization plans. We present a novel solution that integrates machine learning and road network optimization. Our solution consists of two complementary parts. The first part is an efficient algorithm that uses reinforcement learning to find the best road network configurations at real-time. The second part is a dynamic routing mechanism, which helps connected vehicles adapt to the change of the road network. Our extensive experimental results demonstrate that the proposed solution can substantially reduce the average travel time in a variety of scenarios, whilst being computationally efficient and hence applicable to real-life situations.

Deep Neural Networks (DNNs) generally require large-scale datasets for training. Since manually obtaining clean labels for large datasets is extremely expensive, unsupervised models based on domain-specific heuristics can be used to efficiently infer the labels for such datasets. However, the labels from such inferred sources are typically noisy, which could easily mislead and lessen the generalizability of DNNs. Most approaches proposed in the literature to address this problem assume the label noise depends only on the true class of an instance (i.e., class-conditional noise). However, this assumption is not realistic for the inferred labels as they are typically inferred based on the features of the instances. The few recent attempts to model such instance-dependent (i.e., feature-dependent) noise require auxiliary information about the label noise (e.g., noise rates or clean samples). This work proposes a theoretically motivated framework to correct label noise in the presence of multiple labels inferred from unsupervised models. The framework consists of two modules: (1) MULTI-IDNC, a novel approach to correct label noise that is instance-dependent yet not class-conditional; (2) MULTI-CCNC, which extends an existing class-conditional noise-robust approach to yield improved class-conditional noise correction using multiple noisy label sources. We conduct experiments using nine real-world datasets for three different classification tasks (images, text and graph nodes). Our results show that our approach achieves notable improvements (e.g., 6.4% in accuracy) against state-of-the-art baselines while dealing with both instance-dependent and class-conditional noise in inferred label sources.

Intelligent traffic management solutions that leverage machine learning have gained a lot of interest in recent years. These techniques, however, cannot be deployed in real-world settings at a desirable pace due to technological barriers. Thus, easily customizable, realistic simulation environments are needed to train and verify the effectiveness of machine learning algorithms for traffic control. We propose an easily extendable traffic simulation system named e-SMARTS to allow researchers to experiment with novel data-driven traffic management algorithms in a setup that mimics real-world traffic conditions. We demonstrate the flexibility of e-SMARTS using widely researched traffic management solutions for Autonomous Intersection Management (AIM). In the demonstration, we present several pluggable algorithms for AIM and show that these computationally efficient algorithms can achieve effective and safe results.

Graph combinatorial optimization (CO) is a widely studied problem with use-cases stemming from many fields. Typically, in real-world applications, the features of a graph tend to change over time (e.g. traffic congestion, or travel time), thus, finding a solution to the dynamic graph CO problem is critical. In recent years, using deep learning techniques to find heuristic solutions for NP-hard CO problems has gained much interest as these learned heuristics can find near-optimal solutions efficiently. However, most of the existing methods for learning heuristics focus on static CO problems. The dynamic nature makes NP-hard CO problems much more challenging to learn, and the existing methods fail to find reasonable solutions. We propose a novel architecture named Graph Temporal Attention with Reinforcement Learning (GTA-RL) to learn heuristic solutions for dynamic versions of graph CO problems. We then extend our architecture to learn heuristics for the real-time version of CO problems where all input features of a problem are not known a priori, but rather learned in real-time. A detailed experimental evaluation against several state-of-the-art learning-based algorithms and optimal solvers demonstrates the efficiency and effectiveness of our approach.

With increasing connectivity and autonomy in traffic eco-systems, Autonomous Intersection Management (AIM) has attracted strong attention from the research community. AIM helps optimize traffic by coordinating the trajectory of connected vehicles around intersections. Most of the existing AIM solutions are developed for single-objective optimization problems that are focused on improving traffic flow. A complete AIM solution needs to perform bi-objective optimization that considers both traffic flow and safety. However, the computational complexity for achieving both objectives is significantly high with the existing solutions, especially when traffic demand is stochastic. We address the limitations of the existing solutions using deep reinforcement learning (deep RL) that helps solve complex problems efficiently. Our solution uses two types of RL agents. The first type is intersection-level agents, which generate theoretically sound trajectory plans for individual vehicles approaching intersections. The second type is vehicle-level agents that control vehicles' actual trajectories around the intersections based on the plans. Both agents incorporate traffic flow and safety constraints into their decision making. Our experimental results show that our solution achieves a high safety level with a minimum impact on travel time.

Due to the importance of road freight, there is a significant cost of delaying freight vehicles on the road. In this work, we focus on freight vehicle optimization by reducing delays at intersections. Our simulation study evaluates the effectiveness of an autonomous intersection management strategy that prioritizes connected freight vehicles using intelligent traffic lights. We simulate a wide range of traffic scenarios on our microscopic traffic simulator. Our results show that the strategy can help reduce the delay of freight vehicles with a minimal impact on other vehicles in a real road network. Our simulations also reveal the scenarios where the strategy works best and where it should be avoided. Effects of individual parameters are also measured through simulations.

Autonomous intersection management has the ability to reduce congestion at intersections significantly, compared to classical traffic signal control in the era of connected autonomous vehicles. Autonomous intersection management requires time and speed adjustment for vehicles arriving at an intersection for collision-free passing through the intersection. Due to its computational complexity, this problem has been studied only when vehicle arrival times towards the vicinity of the intersection are known beforehand or with other simplifying scenarios which limits the applicability of these solutions for real-time settings. To solve the real-time autonomous traffic intersection management problem, we propose a reinforcement learning (RL) based multiagent architecture and a novel RL algorithm coined multi-discount Q-learning. In multi-discount Q-learning, we introduce a simple yet effective way to solve a Markov Decision Process by preserving both short-term and long-term goals, which is crucial for collision-free speed control. Our experimental results using microscopic simulations show that our RL-based multiagent solution can achieve near-optimal performance efficiently when minimizing the travel time through an intersection.

In agile iterative development, an agile team needs to analyze documented information for effort estimation and sprint planning. While documentation can be changed, the documentation changes after sprint planning may invalidate the estimated effort and sprint plan. Hence, to help the team be aware of the potential documentation changes, we developed DocWarn to estimate the probability that a work item will have documentation changes. We developed three variations of DocWarn, which are based on the characteristics extracted from the work items (DocWarn-C), the natural language text (DocWarn-T), and both inputs (DocWarn-H).

Abstract Story Points (SP) are an effort unit that is used to represent the relative effort of a work item. In Agile software development, SP allows a development team to estimate their delivery capacity and facilitate the sprint planning activities. Although Agile embraces changes, SP changes after the sprint planning may negatively impact the sprint plan. To minimize the impact, there is a need to better understand the SP changes and an automated approach to predict the SP changes. Hence, to better understand the SP changes, we examine the prevalence, accuracy, and impact of information changes on SP changes. Through the analyses based on 19,349 work items spread across seven open-source projects, we find that on average, 10% of the work items have SP changes. These work items typically have SP value increased by 58%-100% relative to the initial SP value when they were assigned to a sprint. We also find that the unchanged SP reflect the development time better than the changed SP. Our qualitative analysis shows that the work items with changed SP often have the information changes relating to updating the scope of work. Our empirical results suggest that SP and the scope of work should be reviewed prior or during sprint planning to achieve a reliable sprint plan. Yet, it could be a tedious task to review all work items in the product (or sprint) backlog. Therefore, we develop a classifier to predict whether a work item will have SP changes after being assigned to a sprint. Our classifier achieves an AUC of 0.69-0.8, which is significantly better than the baselines. Our results suggest that to better manage and prepare for the unreliability in SP estimation, the team can leverage our insights and the classifier during the sprint planning. To facilitate future studies, we provide the replication package and the datasets, which are available online.

Abstract Fault‐tolerance is an essential part of a stream processing system that guarantees data analysis could continue even after failures. State‐of‐the‐art distributed stream processing systems use checkpointing to support fault‐tolerance for stateful computations where the state of the computations is periodically persisted. However, the frequency of performing checkpoints impacts the performance (utilization, latency, and throughput) of the system as the checkpointing process consumes resources and time that can be used for actual computations. In practice, systems are often configured to perform checkpoints based on crude values ignoring factors such as checkpoint and restart costs, leading to suboptimal performance. In our previous work, we proposed a theoretical optimal checkpoint interval that maximizes the system utilization for stream processing systems to minimize the impact of checkpointing on system performance. In this article, we investigate the practical benefits of our proposed theoretical optimal by conducting experiments in a real‐world cloud setting using different streaming applications; we use Apache Flink, a well‐known stream processing system for our experiments. The experiment results demonstrate that an optimal interval can achieve better utilization, confirming the practicality of the theoretical model when applied to real‐world applications. We observed utilization improvements from 10% to 200% for a range of failure rates from 0.3 failures per hour to 0.075 failures per minute. Moreover, we explore how performance measures: latency and throughput are affected by the optimal interval. Our observations demonstrate that significant improvements can be achieved using the optimal interval for both latency and throughput.

Big data frameworks such as Apache Spark is becoming prominent to perform large-scale data analytics jobs in various domains. However, due to limited resource availability, the local or on-premise computing resources are often not sufficient to run these jobs. Therefore, public cloud resources can be hired on a pay-per-use basis from the cloud service providers to deploy a Spark cluster entirely on the cloud. Nevertheless, using only cloud resources can be costly. Hence, both local and cloud resources nowadays are used together to deploy a hybrid cloud computing cluster. However, scheduling jobs in a cluster deployed on hybrid clouds is challenging in the presence of various Service-Level Agreement (SLA) demands such as cost minimization and job deadline guarantee. Most of the existing works either consider a public or a locally deployed cluster and mainly focus on improving job performance in the cluster. In this article, we propose efficient scheduling algorithms that leverage from different VM instance pricing in a hybrid cloud deployed cluster to optimize the Virtual Machine (VM) usage cost for both local and cloud resources and maximize the job deadline met percentage. We have conducted extensive simulation-based experiments to compare our proposed algorithms with the baseline approaches. In addition, we have developed a prototype system on top of Apache Mesos cluster manager and performed real experiments to evaluate the applicability of our proposed approaches in a real platform with benchmark applications. The results show that our proposed algorithms are highly scalable and reduce the cost of VM usage of a hybrid cluster for up to 20 percent.

Big data frameworks such as Spark and Hadoop are widely adopted to run analytics jobs in both research and industry. Cloud offers affordable compute resources which are easier to manage. Hence, many organizations are shifting towards a cloud deployment of their big data computing clusters. However, job scheduling is a complex problem in the presence of various Service Level Agreement (SLA) objectives such as monetary cost reduction, and job performance improvement. Most of the existing research does not address multiple objectives together and fail to capture the inherent cluster and workload characteristics. In this article, we formulate the job scheduling problem of a cloud-deployed Spark cluster and propose a novel Reinforcement Learning (RL) model to accommodate the SLA objectives. We develop the RL cluster environment and implement two Deep Reinforce Learning (DRL) based schedulers in TF-Agents framework. The proposed DRL-based scheduling agents work at a fine-grained level to place the executors of jobs while leveraging the pricing model of cloud VM instances. In addition, the DRL-based agents can also learn the inherent characteristics of different types of jobs to find a proper placement to reduce both the total cluster VM usage cost and the average job duration. The results show that the proposed DRL-based algorithms can reduce the VM usage cost up to 30%.

With the rapid evolution of social media, fake news has become a significant social problem, which cannot be addressed in a timely manner using manual investigation. This has motivated numerous studies on automating fake news detection. Most studies explore supervised training models with different modalities (e.g., text, images, and propagation networks) of news records to identify fake news. However, the performance of such techniques generally drops if news records are coming from different domains (e.g., politics, entertainment), especially for domains that are unseen or rarely-seen during training. As motivation, we empirically show that news records from different domains have significantly different word usage and propagation patterns. Furthermore, due to the sheer volume of unlabelled news records, it is challenging to select news records for manual labelling so that the domain-coverage of the labelled dataset is maximised. Hence, this work: (1) proposes a novel framework that jointly preserves domain-specific and cross-domain knowledge in news records to detect fake news from different domains; and (2) introduces an unsupervised technique to select a set of unlabelled informative news records for manual labelling, which can be ultimately used to train a fake news detection model that performs well for many domains while minimizing the labelling cost. Our experiments show that the integration of the proposed fake news model and the selective annotation approach achieves state-of-the-art performance for cross-domain news datasets, while yielding notable improvements for rarely-appearing domains in news datasets.

Serverless computing enables rapid application development and deployment by composing loosely coupled microservices at a scale. This emerging paradigm greatly unburdens the users of cloud environments, from the need to provision and manage the underlying cloud resources. With this shift in responsibility, the cloud provider faces the challenge of providing acceptable performance to the user without compromising on reliability, while having minimal knowledge of the application requirements. Sub-optimal resource allocations, specifically the CPU resources, could result in the violation of performance requirements of applications. Further, the fine-grained serverless billing model only charges for resource usage in terms of function execution time. At the same time, the provider has to maintain the underlying infrastructure in always-on mode to facilitate asynchronous function calls. Thus, achieving optimum utilization of cloud resources without compromising on application requirements is of high importance to the provider. Most of the current works only focus on minimizing function execution times caused by delays in infrastructure set up and reducing resource costs for the end-user. However, in this paper, we focus on both the provider and user's perspective and propose a function placement policy and a dynamic resource management policy for applications deployed in serverless computing environments. The policies minimize the resource consumption cost for the service provider while meeting the user's application requirement, i.e., deadline. The proposed solutions are sensitive to deadline and efficiently increase the resource utilization for the provider, while dynamically managing resources to improve function response times. We implement and evaluate our approach through simulation using ContainerCloudSim toolkit. The proposed function placement policy when compared with baseline scheduling techniques can reduce resource consumption by up to three times. The dynamic resource allocation policy when evaluated with a fixed resource allocation policy and a proportional CPU-shares policy shows improvements of up to 25% in meeting the required function deadlines.

Effort estimation is an integral part of activities planning in Agile iterative development. An Agile team estimates the effort of a task based on the available information which is usually conveyed through documentation. However, as documentation has a lower priority in Agile, little is known about how documentation effort can be optimized while achieving accurate estimation. Hence, to help practitioners achieve just-enough documentation for effort estimation, we investigated the different types of documented information that practitioners considered useful for effort estimation. We conducted a survey study with 121 Agile practitioners across 25 countries. Our survey results showed that (1) despite the lower priority of documentation in Agile practices, 98% of the respondents considered documented information moderately to extremely important when estimating effort, (2) 73% of them reported that they would re-estimate a task when the documented information was changed, and (3) functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were ranked as the most useful types of documented information for effort estimation. Nevertheless, many respondents reported that these useful types of documented information were occasionally changing or missing. Based on our study results, we provide recommendations for agile practitioners on how effort estimation can be improved by focusing on just-enough documentation.

Traffic forecasting plays a vital role in traffic management systems. Recently, deep learning models have been applied to citywide traffic forecasting. However, the existing work models and predicts traffic at a single (dense) resolution, making it challenging to capture long-range spatial dependencies or high-level traffic dynamics. This shortcoming limits the accuracy of prediction and results in computationally expensive models. We propose a traffic forecasting model based on deep convolutional networks to improve the accuracy of citywide traffic forecasting. Our model uses a hierarchical architecture that captures traffic dynamics at multiple spatial resolutions. Based on this architecture, we apply a multi-task learning scheme, which trains the model to predict traffic at different resolutions. Our model helps provide a coherent understanding of traffic dynamics by capturing spatial dependencies between different regions of a city. Experimental results on multiple real datasets show that our model can achieve competitive results compared to complex state-of-the-art approaches while being more computationally efficient.

Many learning tasks involve multi-modal data streams, where continuous data from different modes convey a comprehensive description about objects. A major challenge in this context is how to efficiently interpret multi-modal information in complex environments. This has motivated numerous studies on learning unsupervised representations from multi-modal data streams. These studies aim to understand higher-level contextual information (e.g., a Twitter message) by jointly learning embeddings for the lower-level semantic units in different modalities (e.g., text, user, and location of a Twitter message). However, these methods directly associate each low-level semantic unit with a continuous embedding vector, which results in high memory requirements. Hence, deploying and continuously learning such models in low-memory devices (e.g., mobile devices) becomes a problem. To address this problem, we present METEOR, a novel MEmory and Time Efficient Online Representation learning technique, which: (1) learns compact representations for multi-modal data by sharing parameters within semantically meaningful groups and preserves the domain-agnostic semantics; (2) can be accelerated using parallel processes to accommodate different stream rates while capturing the temporal changes of the units; and (3) can be easily extended to capture implicit/explicit external knowledge related to multi-modal data streams. We evaluate METEOR using two types of multi-modal data streams (i.e., social media streams and shopping transaction streams) to demonstrate its ability to adapt to different domains. Our results show that METEOR preserves the quality of the representations while reducing memory usage by around 80% compared to the conventional memory-intensive embeddings.

We study the problem of tracking the movement of groups using sparse trajectory data extracted from Location Based Social Networks (LBSNs). Tracking group movement using LBSN data is challenging because the data may contain a large amount of noise due to the lack of stability in group entity, spatial extent and posting time. We propose a first-of-its-kind solution, Group Kalman Filter (GKF), which aims to improve the effectiveness of group tracking by predicting the spatial properties of groups with a group movement model. Our experiments with real LBSN data and synthetic LBSN data show that GKF can detect groups and predict group movement with a high level of accuracy and efficiency.

Each year, millions of people either die or get injured due to road incidents. Thus, integrating safety optimization techniques into future traffic systems is of utmost importance. Emerging connected vehicle technologies have enabled ways to manage traffic networks with optimization goals such as travel time efficiency, fuel efficiency. However, these existing studies have focused less on maximizing traffic safety. Increasing space between vehicles in the road network with an acceptable travel time increase will help to improve the safety of the system. We propose the Platooning Graph, which is capable of modelling the inter-vehicular spacing optimization problem and we provide a fast and readily deployable algorithm to find a good approximate solution. Using microscopic traffic simulations, we demonstrate how the proposed method can improve safety, with minimal impact on travel time.

We present a new unsupervised dimensionality reduction technique, called LN-SNE, for anomaly detection. LN-SNE generates a parametric embedding by means of Restricted Boltzmann Machines and uses a heavy-tail distribution to project data to a lower dimensional space such that dissimilarities between normal data and anomalies are preserved or strengthened. We compare LN-SNE to several benchmark dimensionality reduction methods on real datasets. The results suggest that LN-SNE for anomaly detection is less sensitive to the dimension of the latent space than the other methods and outperforms them in terms of accuracy. We empirically show that our technique scales near-linearly with respect to the number of dimensions and data size.

Compared with traditional news media, social media nowadays provides a richer and more timely source of news. We are interested in multi-spatial level event detection from geo-tagged tweet streams. Specifically, in this paper we (1) examine the statistical characteristic for the time series of the number of geo-tagged tweets posted from specific regions during a short time interval, e.g., one minute; (2) verify from over thirty datasets that while almost all such time series exhibit self-similarity, those that correspond to events, especially short-term and unplanned outbursts, follow a power-law distribution; (3) demonstrate that these findings can be applied to facilitate event detection from tweet streams. We propose two algorithms-Power-law basic and Power-law advanced, where Power-law basic only checks the existence of power-law distributions in the time series from tweet streams at multi-spatial scales, without looking into the content of each tweet, and Power-law advanced integrates power-law verification with semantic analysis via word embedding. Our experiments on multiple datasets show that when combined with a Quad-tree, the seemingly naive algorithm of Power-law basic achieves comparable results with more advanced event detection methods, while the semantic analysis enhanced version, Power-law advanced, can significantly increase both the precision and the recall.

Identifying events happening in a specific locality is important as an early warning for accidents, protests, elections or breaking news. However, this location-specific event detection is challenging as the locations and types of events are not known beforehand. To address this problem, we propose an online spatio-temporal event detection system using social media that is able to detect events at different time and space resolutions. First, we exploit a quad-tree method to split the geographical space into multiscale regions based on the density of social media data. Then, we implement a statistical unsupervised approach using Poisson distribution and a smoothing method for highlighting regions with unexpected density of social posts. Further, event duration is estimated by merging events happening in the same region at consecutive time intervals. A post processing stage is introduced to filter out events that are spam, fake or wrong. Finally, we incorporate simple semantics by using social media entities to assess the integrity, and accuracy of detected events. The proposed method is evaluated using Twitter and Flickr for the city of Melbourne based on recall and precision measures. We also propose a new quality measure named strength index, which automatically measures how accurate the reported event is.

Currently, the enormous span of social media usage - while providing valuable resources for linguistic behavior analysis - makes tracking and understanding these multilingual discussions a challenging task. We have undertaken a multidisciplinary comprehensive study of multilingual discussions via the development of specialized data collection techniques that discover and track multilingual users of social media, and their associated discussions, within a defined geographical region. To facilitate automatic discussion analysis of large numbers of discussions we generated a machine learning model based on ground truth data obtained from Amazon Turk. Our approach goes beyond analyzing social media posts in isolation, by analyzing them in the context of the discussion in which they appear. We show a selection of example discussions found using our approach which reveals a number of interesting socio-linguistic interactions in the communities that we sampled, in support of approach as a general methodology for multilingual community analysis.

Building spatiotemporal activity models for people's activities in urban spaces is important for understanding the ever-increasing complexity of urban dynamics. With the emergence of Geo-Tagged Social Media (GTSM) records, previous studies demonstrate the potential of GTSM records for spatiotemporal activity modeling. State-of-the-art methods for this task embed different modalities (location, time, and text) of GTSM records into a single embedding space. However, they ignore Non-GeoTagged Social Media (NGTSM) records, which generally account for the majority of posts (e.g., more than 95% in Twitter), and could represent a great source of information to alleviate the sparsity of GTSM records. Furthermore, in the current spatiotemporal embedding techniques, less focus has been given to the users, who exhibit spatially motivated behaviors. To bridge this research gap, this work proposes USTAR, a novel online learning method for User-guided SpatioTemporal Activity Representation, which (1) embeds locations, time, and text along with users into the same embedding space to capture their correlations; (2) uses a novel collaborative filtering approach to incorporate both NGTSM and GTSM records in learning; and (3) introduces a novel sampling technique to learn spatiotemporal representations in an online fashion to accommodate recent information into the embedding space, while avoiding overfitting to recent records and frequently appearing units in social media streams. Our results show that USTAR substantially improves the state-of-the-art for region retrieval and keyword retrieval and its potential to be applied to other downstream applications such as local event detection.

Real-time processing of large-scale, time-varying graphs is becoming essential in bigdata streaming applications that require high-throughput and low-latency. Streaming applications typically perform graph computations on periodic snapshots of the graph, using windowing. For large-scale graphs, that contains millions of edges, distributed parallel processing is essential to meet the throughput and latency needs of streaming applications. Apache Storm, a widely used stream processing system, provides a topology-based processing architecture which supports distribution and pipelining to meet the latency and throughput needs; a typical Storm topology contains a fixed number of stages, each stage with a single operator supporting a single computation step, which can be parallelized through horizontal scaling (bolt instances). One of the limitations of such a topology, is the fixed number of stages, which makes it not suitable for graph algorithms that require iterative computations, such as the PageRank algorithm, because the number of iterations, hence the number of stages in the topology, is unknown a-priori. In this paper we explore how iterative graph algorithms can be executed efficiently in a distributed manner for scalability (with the graph distributed across a set of Storm bolts), using a single stage of a Storm topology, using Storm's cyclic communication (S-CC) mechanism. We show that S-CC approach outperforms alternative approaches that can be used for parallel-iterative computations in Storm, such as off-loading to a third-party graph processing system such as Giraph or using a distributed memory architecture, both in terms of scalability and performance.

Stream processing systems are used for a plethora of low-latency applications that deal with high volumes and varieties of data. In general, streaming applications are formulated as a fixed number of inter-connected operators, where the operator graph indicates the sequence of computations that apply to the streaming data in motion. Typically an iterative operation in a streaming application is done by embedding the iteration in a single operator (which excludes multiple operator instances and hence not scalable) or by using a sequence of operators, one for each iteration (which requires knowledge of the number of iterations a priori). However, the number of iterations of some applications such as k-means clustering cannot be determined at the creation of the application as it depends on the properties of the data received at runtime and the convergence criterion. Hence only a limited number of iterative computations can be executed using this approach. In this paper, we propose a communication model to support intra-operator communication so that iterative computations, including those with an arbitrary number of iterations, can be efficiently executed in streaming applications. We show that the proposed model can support different iterative algorithms that have complex communication patterns. Finally, through evaluating a number of parallel iterative algorithms using large-scale datasets, we demonstrate the scalability and performance of our proposed communication model and compare it to the existing approaches used for constructing iterative streaming applications.

There are many real world applications that require to identify movement of users such as identifying movement corridors, most popular paths, and nearest neighbours. If one is not given trajectories mapping to movement of people but rather sporadic location data, such as location based social network data, finding movement related information becomes difficult. Rather than processing all points in a data set given a query, a clever approach is to construct a graph, based on user locations, and query this graph for all queries. One example is the shortest path graph. However the shortest path graph can be inefficient and ineffective analysing movement, as it calculates the graph considering all points in a data set. We propose the stepping stone graph, which calculates graph considering point pairs rather than all points, that focuses on local possible movement, making it both efficient and effective for location based social network related queries. We demonstrate its uses by applying it in the aforementioned domain and comparing with the shortest path graph. We also compare its properties to a range of other graphs.

Microscopic traffic simulators play a major role to carry research on transportation problems. Microscopic traffic simulation is powerful because it enables efficient analysis of complex traffic problems to the highest level of detail. We developed Scalable Microscopic Adaptive Road Traffic Simulator (SMARTS) [14] that can perform large-scale simulations at a high speed by utilizing distributed computing resources. Previous results show that SMARTS can run 1.14 times faster than real time when simulating one million vehicles for the city of Melbourne on 30 distributed processors, while producing highly accurate simulation results. SMARTS' pluggable architecture allows it to be easily extended to simulate specific scenarios of interest to users. In this demonstration we show how SMARTS can be used to simulate an intersection design, the P-turn, in a major intersection of Melbourne. Our simulation shows the impact of the design on the traffic flow, confirming the justification for introduction of the particular intersection. The demo can be used as a template for future use of the simulator for other traffic problems.

Data Stream Processing (DSP) is a widely used programming paradigm to process an unbounded event stream. Often, DSP frameworks are deployed on the cloud with a scalable resource model. One of the key requirements of DSP is to produce results with low latency. With the emergence of IoT, many event sources have been located outside the cloud which can result in higher end-to-end latency due to communication overhead. However, due to the abundance of resources at the IoT layer, Edge computing has emerged as a viable computational paradigm. In this paper, we devise an optimisation framework, consisting of a constraint satisfaction formulation and a system model, that aims to minimise end-to-end latency through appropriate placement of DSP operators either on cloud nodes or edge devices, i.e. deployed in an edge-cloud integrated environment. We test our optimisation framework using OMNeT++, with realistic topologies and power consumption data, and show that it is capable of achieving approx 1.65 times reduction of latency compared to edge-only and cloud-only placements, which in turn also reduces the energy consumption per event by up to approx 4% at the edge layer. To the best of our knowledge our optimisation framework is the first of its kind to integrate power, bandwidth and CPU constraints with latency minimisation.

One-class support vector machines (OCSVMs) are very effective for semisupervised anomaly detection. However, their performance strongly depends on the settings of their hyperparameters, which has not been well studied. Moreover, unavailability of a clean training set that only comprises normal data in many real-life problems has given rise to the application of OCSVMs in an unsupervised manner. However, it has been shown that if the training set includes anomalies, the normal boundary created by OCSVMs is prone to skew toward the anomalies. This problem decreases the detection rate of anomalies and results in poor performance of the classifier. In this paper, we propose a new technique to set the hyperparameters and clean suspected anomalies from unlabelled training sets. The proposed method removes suspected anomalies based on a $K$ -nearest neighbors technique, which is then used to directly estimate the hyperparameters. We examine several benchmark data sets with diverse distributions and dimensionality. Our findings suggest that on the examined data sets, the proposed technique is roughly 70 times faster than supervised parameter estimation via grid-search and cross validation, and one to three orders of magnitude faster than broadly used semisupervised and unsupervised parameter estimation methods for OCSVMs. Moreover, our method statistically outperforms those semisupervised and unsupervised methods and its accuracy is comparable to supervised grid-search and cross validation.

With the proliferation of sensors and the increased connectivity of citizens, many global cities are increasingly adopting Smart City initiatives. Such initiatives provide real-time monitoring capabilities, and effective modelling techniques allow the prediction of future states in a city. For example, urban electricity smart meter data can be utilised to predict future demand in order to facilitate capacity planning. However, the accuracy of this foresight is often marred by low quality and missing sensor data in real-world systems. In this work, we focus on the problem of reliable forecasting by mitigating the effect of missing data on forecast accuracy. In order to mitigate the effects of missing data, we develop a multi-task learning scheme to jointly learn Gaussian Process Regression models between highly correlated sensors. We demonstrate that our methods are robust in a variety of error generation scenarios. We validate our methods based on publicly available and real-world datasets related to electricity smart meters in a university campus and pedestrian counts in a global city, where we achieve significant improvement over competitive baselines and other effective forecasting methods.

Twitter is a popular microblogging service, where users frequently engage in discussions about various topics of interest, ranging from popular topics (e.g., music) to niche topics (e.g., politics). With the large amount of tweets, a key challenge is to automatically model and determine the discussion topics without having prior knowledge of the types and number of topics, or requiring the technical expertise to define various algorithmic parameters. For this purpose, we propose the Clustering-based Topic Modelling (ClusTop) algorithm that constructs various types of word network and automatically determines the discussion topics using community detection approaches. Unlike traditional topic models, ClusTop is able to automatically determine the appropriate number of topics and does not require numerous parameters to be set. The ClusTop algorithm is also able to capture the syntactic meaning in tweets via the use of bigrams, trigrams and other word combinations in constructing the word network graph. Using three Twitter datasets with labelled crises and events as topics, ClusTop has been shown to outperform various baselines in terms of topic coherence, pointwise mutual information, precision, recall and F-score.

Topic modelling is a well-studied field that aims to identify topics from traditional documents such as news articles and reports. More recently, Latent Dirichlet Allocation (LDA) and its variants, have been applied on social media platforms to model and study topics relating to sports, politics and companies. While these applications were able to successfully identify the general topics, we posit that standard LDA can be augmented with spatial and temporal considerations based on the geo-coordinates and timestamps of social media posts. Towards this effort, we propose a spatial and temporal variant of LDA to better detect more specific topics, such as a particular art exhibit held at a museum or a security incident happening on a particular day. We validate our approach on a Twitter dataset and find that the detected topics are well-aligned to real-life events happening on the specific days and locations.

Large-scale data processing framework like Apache Spark is becoming more popular to process large amounts of data either in a local or a cloud deployed cluster. When an application is deployed in a Spark cluster, all the resources are allocated to it unless users manually set a limit on the available resources. In addition, it is not possible to impose any user-specific constraints and minimize the cost of running applications. In this paper, we present dSpark, a lightweight, pluggable resource allocation framework for Apache Spark. In dSpark, we have modelled the application completion time with respect to the number of executors and application input/iteration. This model is further used in our proposed resource allocation model where a deadlinebased, cost-efficient resource allocation scheme can be selected for any application. As opposed to the existing frameworks that focus more on modelling the number of VMs to use for an application, we have modelled both the application cost and completion time with respect to executors, hence providing a finegrained resource allocation scheme. In addition, users do not need to specify any application types in dSpark. We have evaluated our proposed framework through extensive experimentation, which shows significant performance benefits. The application completion time prediction model has a mean relative error (RE) less than 7% for different types of applications. Furthermore, we have shown that our proposed resource allocation model minimizes the cost of running applications and selects effective resource allocation schemes under varying user-specific deadlines.

Traffic congestion is a serious problem that is only expected to get worse in the future. Statistics shows that half of traffic congestion is caused by temporary disruptions like accidents. These events have dramatic impact on road network availability and cause huge delays for commuters. Also, they are usually unexpected and hard to manage by traffic authorities. State-of-the-art navigation systems started to provide real-time information about traffic conditions to help users make better routing decisions. However, traffic in the road network changes rapidly and the advice calculated now may not be valid after few minutes. This is especially critical in the presence of traffic incidents, where the impact of the incident could cause traffic to propagate to nearby roads. Thus, it is important for navigation systems to consider the evolution and future impact of traffic events. In this work, we present a navigation system that uses faster than realtime simulations to predict the evolution of traffic events and help drivers proactively avoid congestion caused by events. The system can subscribe to real-time traffic information and forecast the traffic conditions using fast simulations. We evaluate our approach through extensive experiments to test the performance and accuracy of the simulator with real data obtained from TomTom Traffic API. Also, we test the quality of navigation advice in realistic settings and show that our solution is able to help drivers avoid congested areas in cases where even real-time update methods lead drivers to congested routes.

Distributed spatial simulations commonly employ Bulk Synchronous Parallel model (BSP) implementation. However, implementations using BSP are usually fraught with the straggler problem, where the delay of any worker slows down the entire system. Random stragglers commonly occur due to many reasons: imbalanced workload, operating system scheduling, or communication delays. The straggler problem is further exasperated with increasing parallelism. To reduce the straggler problem and preserve simplicity and scalability advantages of the BSP model, we propose a new parallel model, which we call Priority Asynchronous Parallel (PAP) model. PAP exploits data dependencies of parallel processes to be computed and synchronized based on data priority to the other workers. For further computational improvement, we develop a load balancing and partitioning method, called GridGraph that utilizes the spatial and connectivity properties of the simulation space to reduce the size of exchanged data in addition to balancing the workload among workers. The proposed schemes are implemented and evaluated in a microscopic traffic simulator. Running traffic simulation for Melbourne, Beijing, and New York cities on 80 workers, the simulation achieves a performance speedup of around 47.4% for Melbourne, 52.18% for Beijing, and 65.84% for New York, using PAP model combined with GridGraph partitioning compared to BSP model.

The dynamic nature of real-world networks, such as social networks and communication networks, has increased the focus towards real-time dynamic graph analysis. Observations made based on real-time analysis of dynamic graphs reflect the latest properties of the graph and have the most value in real-time analysis. Computing graph properties of large-scale, fast-evolving graphs in real-time is challenging due, not only to the high computational and memory cost, but also to the understanding of the result with respect to the data from which it was derived. This paper proposes a multi-stage hierarchical window model that can aid in rigorous understanding of complicated real-time results and we apply it to generate graphs based on real-time updates along with periodic computations on graph snapshots for processing dynamic graphs. Moreover, the paper discusses the utilization of parallel window computation. The paper evaluates the hierarchical model through analyzing graphs formed by cooccurring hashtags in a Twitter data-stream.

Running large-scale behavioral simulations requires high computational power, which can be acquired by distributing computation workload to multiple computing nodes (i.e., workers) that run in parallel. The implementations of such systems commonly follow the Bulk Synchronous Parallel (BSP) model. However, implementations using BSP usually suffer from the straggler problem, where the delay of any worker slows down the entire simulation. The problem usually occurs due to communication delays or imbalanced workload among workers. To mitigate the straggler problem, we propose a novel parallel computational model, called Priority Synchronous Parallel (PSP) model. PSP exploits data dependencies of parallel processes to determine high priority data to be computed and synchronized while computing the remaining data. PSP is implemented and evaluated using traffic simulations for three large cities. The proposed technique shows significant performance improvements over the BSP model.

Apache Storm is a fault-tolerant, distributed inmemory computation system for processing large volumes of high-velocity data in real-time. As an integral part of the fault-tolerance mechanism, Storm's state management is achieved by a checkpointing framework, which commits states regularly and recovers lost states from the latest checkpoint. However, this method involves a remote data store for state preservation and access, resulting in significant overheads to the performance of error-free execution.In this paper, we propose E-Storm, a replication-based state management system that actively maintains multiple state backups on different worker nodes. We build a prototype on top of Storm by extending it with monitoring and recovery modules to support inter-task state transfer whenever needed. The experiments carried out on synthetic and real-world streaming applications confirm that E-Storm outperforms the existing checkpointing method in terms of the resulting application performance, obtaining as much as 9.44 times throughput improvement while reducing the application latency down to 9.8%.

In time-series forecasting, regression is a popular method, with Gaussian Process Regression widely held to be the state of the art. The versatility of Gaussian Processes has led to them being used in many varied application domains. However, though many real-world applications involve data which follows a working-week structure, where weekends exhibit substantially different behavior to weekdays, methods for explicit modelling of working-week effects in Gaussian Process Regression models have not been proposed. Not explicitly modelling the working week fails to incorporate a significant source of information which can be invaluable in forecasting scenarios. In this work we provide novel kernel-combination methods to explicitly model working-week effects in time-series data for more accurate predictions using Gaussian Process Regression. Further, we demonstrate that prediction accuracy can be improved by constraining the non-convex optimization process of finding optimal hyperparameter values. We validate the effectiveness of our methods by performing multi-step prediction on two real-world publicly available time-series datasets - one relating to electricity Smart Meter data of the University of Melbourne, and the other relating to the counts of pedestrians in the City of Melbourne.

Recently many navigation services have started to collect traffic data from drivers to enhance their services. The focus of their efforts are on real-time estimation of traffic conditions to offer better navigation services and help people avoid traffic incidents. However, using drivers as mere traffic sensors and aggregating their data disposes off any human knowledge and judgment that exists in their choices. Commuters in a city have a valuable knowledge about their city traffic patterns as a result of their daily commute over many years. In our work we take a different approach to navigation advice, we use individual driver trajectories to deduce travel choices. Our goal is to help drivers avoid traffic events by providing routes that bypassed congested routes during similar events in the past. We present a solution that combines graph search and trajectory search to find the fastest path taken in the past using historical trajectories under similar conditions in a city. We evaluate our approach through extensive experiments in realistic settings and experimental results show that our solution was able to find faster paths, avoiding the congested areas in cases where state-of-the-art methods resulted in drivers leading to routes through congested areas.

Similarity based clustering (SBC) overlays are decentralized networks of nodes on the Internet edge, where each node maintains some number of direct connections to other nodes that are most "similar" to it. The challenge is: how do the nodes in the overlay converge to and maintain the most similar neighbors, given that the network is decentralized, is subject to churn and that similarity varies over time. Protocols that simultaneously provide fast convergence and low bandwidth consumption are the objective of this research. We present a protocol, that we call Weighted Benefit Scheme (WBS), that improves upon existing state-of-the-art in this area: it has equivalent convergence rate to the Optimum Benefit Protocol while simultaneously handling churn competitively to the Vicinity protocol. We use real world datasets from Yahoo WebScope that comprises of 15,400 users with 354,000 ratings about 1000 songs and our experiments are performed on the simulation test-bed PeerNet.

Personalized itinerary recommendation is a complex and time-consuming problem, due to the need to recommend popular attractions that are aligned to the interest preferences of a tourist, and to plan these attraction visits as an itinerary that has to be completed within a specific time limit. Furthermore, many existing itinerary recommendation systems do not automatically determine and consider queuing times at attractions in the recommended itinerary, which varies based on the time of visit to the attraction, e.g., longer queuing times at peak hours. To solve these challenges, we propose the PersQ algorithm for recommending personalized itineraries that take into consideration attraction popularity, user interests and queuing times. We also implement a framework that utilizes geo-tagged photos to derive attraction popularity, user interests and queuing times, which PersQ uses to recommend personalized and queue-aware itineraries. We demonstrate the effectiveness of PersQ in the context of five major theme parks, based on a Flickr dataset spanning nine years. Experimental results show that PersQ outperforms various state-of-the-art baselines, in terms of various queuing-time related metrics, itinerary popularity, user interest alignment, recall, precision and F1-score.

Emergency vehicle prioritization is important to the efficiency of emergency services. To address certain challenges in emergency vehicle prioritization, we perform microscopic simulations of an intelligent transportation system, where emergency vehicles broadcast certain information about their routes to nearby vehicles and traffic lights. Our study shows that broadcasting the route information can help reduce the response time of emergency vehicles significantly. In certain case, travel time of emergency vehicles can be as low as 37.1% of that of non-priority vehicles.

Microscopic traffic simulators are important tools for studying transportation systems as they describe the evolution of traffic to the highest level of detail. A major challenge to microscopic simulators is the slow simulation speed due to the complexity of traffic models. We have developed the Scalable Microscopic Adaptive Road Traffic Simulator (SMARTS), a distributed microscopic traffic simulator that can utilize multiple independent processes in parallel. SMARTS can perform fast large-scale simulations. For example, when simulating 1 million vehicles in an area the size of Melbourne, the system runs 1.14 times faster than real time with 30 computing nodes and 0.2s simulation timestep. SMARTS supports various driver models and traffic rules, such as the car-following model and lane-changing model, which can be driver dependent. It can simulate multiple vehicle types, including bus and tram. The simulator is equipped with a wide range of features that help to customize, calibrate, and monitor simulations. Simulations are accurate and confirm with real traffic behaviours. For example, it achieves 79.1% accuracy in predicting traffic on a 10km freeway 90 minutes into the future. The simulator can be used for predictive traffic advisories as well as traffic management decisions as simulations complete well ahead of real time. SMARTS can be easily deployed to different operating systems as it is developed with the standard Java libraries.

With the emergence of data streaming applications that produce large data in motion, anomaly detection in non-stationary environments has become a major research focus. Unknown and unstable behaviour of data over time, limits the application of traditional anomaly detection methods that have been designed for stationary data. Moreover, basic assumptions of many existing works in the adaptive anomaly detection domain, such as the availability of labelled data over time or dealing with a known type of change in the data, are not valid for real-life applications. In this paper, we propose an unsupervised ensemble based anomaly detection method using One Class Support Vector Machines (OCSVMs). The proposed method is able to detect potential changes in the distribution of the normal data and adapts itself accordingly, without requiring any external feedback, e.g., ground truth labels. Moreover, it is able to automatically select an appropriate set of recent instances during learning phases, and an appropriate set of models during prediction phases by identifying active concepts. We evaluate our proposed method against state-of-the-art adaptive anomaly detection methods that can be applied in an unsupervised manner, on both real and synthetic non-stationary data. The results show that with considerably lower computational cost, our method outperforms the other methods.

The problem of unsupervised anomaly detection arises in awide variety of practical applications. While one-class sup-port vector machines have demonstrated their effectiveness asan anomaly detection technique, their ability to model largedatasets is limited due to their memory and time complexityfor training. To address this issue for supervised learning ofkernel machines, there has been growing interest in randomprojection methods as an alternative to the computationallyexpensive problems of kernel matrix construction and sup-port vector optimisation. In this paper we leverage the theoryof nonlinear random projections and propose the RandomisedOne-class SVM (R1SVM), which is an efficient and scalableanomaly detection technique that can be trained on large-scale datasets. Our empirical analysis on several real-life andsynthetic datasets shows that our randomised 1SVM algo-rithm achieves comparable or better accuracy to deep autoen-coder and traditional kernelised approaches for anomaly de-tection, while being approximately 100 times faster in train-ing and testing

Major navigation companies have resorted to crowdsourcing for obtaining traffic data to improve their services. In fact, a new generation of navigation systems are emerging for the sole purpose of connecting drivers on the road (Waze.com) so that they benefit from each other. The focus of these efforts is on real-time traffic estimation. While obtaining real-time data is desirable and has great benefits, it is still limited, given the potential of the detailed individual data from drivers. For example, commuters in a city have valuable knowledge about their city as a result of their daily commute over many years. Using drivers as mere traffic sensors and aggregating their data desposes off any human knowledge that exists in their choices. In this work, we demonstrate a social navigation system that can help drivers avoid road incidents using expert advice in real-time without explicitly consulting drivers but through an automated system built on top of a location-based social network. In our system, we do not aggregate data from individual users and we instead outsource the task of finding a route that best avoids incidents by looking up drivers' old choices.

Making sense of Twitter data streams is challenging due to the extremely high volume of data. One way to address this challenge is to consider these data streams as containing a set of high-level topics. In this research we address the problem of: given a collection of tweets about K high-level topics, how to find topic words that describe these topics as well as how to label each tweet with one of the K topics using a topic modeling approach. Current research has shown that applying topic modeling algorithms directly on tweets does not lead to good results. Hence one approach is to group related tweets together, so as to form a single “pseudo-document”, which is more informative than a single tweet. In this paper we evaluate different grouping schemes found in the literature and propose a new grouping scheme utilizing named entities and word collocations. Results show that our proposed scheme performs better than the existing approaches, to a some extent for all the test cases, and for both finding high-level topics and tweet labeling.

Evolvable Takagi-Sugeno (T-S) models are fuzzy-rule-based models with the ability to continuously learn and adapt to incoming samples from data streams. The model adjusts both premise and consequent parameters to enhance the performance of the model. This paper introduces a new methodology for the estimation of the premise parameters in the evolvable T-S (eTS) model. Incremental updates for the weighted sample mean and inverse of the covariance matrix enable us to construct an evolvable fuzzy rule base that is used to detect outliers and regime changes in the input stream. We compare our model with Angelov's eTS+ model with artificial and real data.

The tracking of microblog discussion, on a given topic, is useful for a wide range of higher level applications. Microblog services like Twitter provide a simple keyword based tracking capability, where any tweet containing a keyword is returned. Due to the short length of microblog posts, using a small number of topic specific query words for tracking, would impact recall. Use of a larger number of keywords (compared to regular document retrieval) is generally required in order to obtain good recall, but this would result in a large number of off-topic posts, resulting in low precision. In our work, we consider the scenario of using a large number of query terms to maintain high recall, for automated tracking of a microblog streams. The challenge we address is how to score each of the returned microblogs, with respect to the query, on-line, in an unsupervised manner, so as to identify those that are on topic. To this end, we proposed a new term-scoring expression, which we call Adjusted Information Gain (AIG), and we compare this to other term-scoring expressions: inverse document frequency, Dice, Jaccard and keyword frequency. Our comparisons consider a selection of document-scoring functions applied to roughly 40 million tweets collects over a 20 day period for each of two topics. Our results show significant improvements (from 8%-40% of the area under the ROC curves) to existing term-scoring expressions, depending on topic and specificity, and provide insight into further work in query expansion techniques.

Increased levels of particulate matter (PM) in the atmosphere have contributed to an increase in mortality and morbidity in communities and are the main contributing factor for respiratory health problems in the population. Currently, PM concentrations are sparsely monitored; for instance, a region of over 2200 square kilometers surrounding Melbourne in Victoria, Australia, is monitored using ten sensor stations. This paper proposes to improve the estimation of PM concentration by complementing the existing high-precision but expensive PM devices with low-cost lower precision PM sensor nodes. Our evaluation reveals that local PM estimation accuracies improve with higher densities of low-precision sensor nodes. Our analysis examines the impact of the precision of the lost-cost sensors on the overall estimation accuracy.

ABSTRACT Two new incremental models for online anomaly detection in data streams at nodes inwireless sensor networksare discussed. These models are incremental versions of a model that uses ellipsoids to detect first, second, and higher‐ordered anomaliesin arrears. The incremental versions can also be used this way but have additional capabilities offered by processing data incrementally as they arrive in time. Specifically, they can detect anomalies ‘on‐the‐fly’ in near real time. They can also be used to track temporal changes in near real‐time because of sensor drift, cyclic variation, or seasonal changes. One of the new models has a mechanism that enables graceful degradation of inputs in the distant past (fading memory). Three real datasets from single sensors in deployed environmental monitoring networks are used to illustrate various facets of the new models. Examples compare the incremental version with the previous batch and dynamic models and show that the incremental versions can detect various types of dynamic anomalies in near real time. Copyright © 2012 John Wiley & Sons, Ltd.

The usefulness of wireless mesh networks (WMNs) highly depends on the effective usage of frequency spectrum, scalability and energy efficiency of the underlying technologies [1]. While maximizing network throughput for given frequency spectrum, and scalability have been extensively studied in multi-hop environments, energy efficiency has not received due attention. In this paper, we propose an enhanced pseudo random access (EPRA) scheme based on the IEEE 802.16-2004 distributed mesh framework to improve energy efficiency. Our proposal addresses issues that we identify with the existing election based transmission timing (EBTT) mechanism of IEEE 802.16-2004. Our extensive simulation results show a very significant improvement in the reception energy efficiency by up to 85%. Moreover, notable improvements are also obtained in the transmission energy efficiency. While we focus mainly on the aspect of energy efficiency, the proposed enhancement also helps to improve the spatial reuse of the frequency spectrum.

Mobility is a challenging issues in mobile networks which significantly impacts the performance of a variety of network protocols. Understanding user motion behavior can improve the performance of mobile network protocols in different aspects. Therefore, we consider how to analyze the motion behavior of mobile nodes in various environments. In this paper we propose a few mobility metrics useful for analysis of individual, collective and geographical behavior of mobile nodes and a simple supervised mobility pattern recognition method which is able to classify mobility traces into different mobility model classes using our proposed mobility metrics. The remaining challenge is to find appropriate features, extracted from mobility traces, which are able to distinguish between different motion behaviors with maximum accuracy and minimum overhead. We propose a method to find the most suitable feature sets for classification of different mobility traces with different motion behaviors. Simulation results show significant performance of our proposed mobility pattern recognition method using appropriate mobility metrics and feature sets.

Recently, there has been focus on augmenting Delay/Disruption Tolerant Networks (DTNs) with easily deployable stationary relay nodes making an unconnected infrastructure to facilitate the data delivery by increasing forwarding opportunities. Relay nodes are capable of downloading, storing, and forwarding the data messages from/to the mobile nodes. Placing the relay nodes is an important issue in DTNs as the performance of the network is dependent to their positions. Relay placement is an NP-hard problem hence it makes it a more complicated issue in DTNs. Existing works in the literature are based on simulation which are suffering from computational complexities dictated by simulation. Moreover, they are optimizing the relay placement only based on specific scenarios. In this paper, we propose a generic analytical model in order to evaluate the performance of DTNs in presence of relay nodes. Our model is dependent on the mobile nodes' mobility pattern, and we consider the case when the mobile nodes move according to the random waypoint model. In order to use the proposed model for placing the relays efficiently, we utilize two heuristic approaches. The first approach is based on optimization of the network performance using simulated annealing and the second one relies on a greedy approach to find the best location for each relay one at a time. Our simulation results show that our approaches outperform the simulation based approaches in terms of data delivery performance.

Gossip-based protocols are one of the popular approaches to estimate aggregates in large scale distributed systems. In this paper, we propose a gossip-based asynchronous and robust protocol to compute aggregates. In our protocol, each node employs the proposed pessimistic approach to exchange its local estimate with one of its neighbors and also uses the proposed recovery scheme which conserves the system mass in the presence of asynchrony and churn. The simulation results presented in this paper show that our protocol yields accurate aggregates in comparatively lesser time.

Wireless Sensor Networks (WSNs) provide a low cost option for gathering spatially dense data from different environments. However, WSNs have limited energy resources that hinder the dissemination of the raw data over the network to a central location. This has stimulated research into efficient data mining approaches, which can exploit the restricted computational capabilities of the sensors to model their normal behavior. Having a normal model of the network, sensors can then forward anomalous measurements to the base station. Most of the current data modeling approaches proposed for WSNs require a fixed offline training period and use batch training in contrast to the real streaming nature of data in these networks. In addition they usually work in stationary environments. In this paper we present an efficient online model construction algorithm that captures the normal behavior of the system. Our model is capable of tracking changes in the data distribution in the monitored environment. We illustrate the proposed algorithm with numerical results on both real-life and simulated data sets, which demonstrate the efficiency and accuracy of our approach compared to existing methods.

Accurately locating the geographical position of Internet hosts has many useful applications. Existing approaches for host geolocation use Internet latency measurements, IP-to-location mapping and also geographical and demographical hints. In this paper, we investigate the applicability of the Maximum Likelihood Estimation (MLE) technique for Internet host geolocation. Our approach is based on a probability model for latency measurements that we developed by analyzing a large set of data collected on the PlanetLab network test bed. This approach uses latency measurements from multiple hosts of known location to the host to be geolocated, to estimate the target location. Using both simulated and real data, we analyze the accuracy of our approach. Our results for geolocating Internet hosts in North America confirms the validity of using MLE with certainty as its accuracy is found to be better in comparison to existing techniques that are based on Internet latency.

This paper presents an eResearch tool for evaluating and comparing Chemical, Biological, Radiological and Nuclear (CBRN) source-backtracking algorithms. The tool enables users to define a CBRN threat scenario to be simulated, which includes the geographical area of concern specified through a map, the simulated threat sources and their placement, and the configuration of sensors. Users can define multiple searchers who act as clients that interact with a central server and search for the simulated radiological source or sources. Based on the simulated measurements received from the searchers, and a chosen source backtracking algorithm, the central server updates the source estimates and issues control vectors to guide each searcher.

IEEE 802.16 based wireless mesh networks (WMNs) are a promising broadband access solution to support flexibility, cost effectiveness and fast deployment of the fourth generation infrastructure based wireless networks. Reducing the time for channel establishment is critical for low latency/interactive applications. In this paper, we propose two modifications to the election based transmission timing (EBTT) mechanism used in the coordinated distributed scheduling (CDS) within the IEEE 802.16 standard, which makes use of the standardized signaling messages and formats. Simulation results show that our proposed modifications results in significantly reduced channel establishment delay - up to 75% with low neighborhood densities - over the standard, and at least 10% with high neighborhood densities. Moreover, the proposed mechanisms are also applicable for the three-hop extended neighborhood CDS when required.

With the success of the peer-to-peer systems, there is an emerging trend to develop an aggregation service that can provide global information about these distributed and decentralized overlays. Existing aggregation protocols have focused on providing an efficient and cost effective solution to compute aggregates. In this paper, we are interested to search the node based on an aggregate query over rapidly changing data e.g., to seek a node with minimum/maximum workload in the network. Solution to such queries can benefit number of applications such as, node-ranking, reputation building, resource scheduling, etc. Here we propose a novel protocol to solve such queries. Our proposed protocol is reactive and pull-based in nature. It intrinsically builds and manages a routing table which routes aggregate query to the desired node in the system. We evaluated our protocol for accuracy, efficiency and communication cost through detailed simulations. We also discuss effects of an explicit routing table management scheme and show that this explicit scheme causes our protocol to work better in presence of churn and node failures.

The large scale and dynamic nature of a distributed system makes it difficult to collect the attributes of the individual nodes in the network. In these systems, often an aggregate (e.g. AVG, COUNT, MIN, MAX, SUM etc) of these attributes is adequate. Epidemic-style protocols are one of the popular approaches to estimate aggregates in such systems. In existing epidemic-style aggregation protocols the accuracy of the estimated aggregate at local nodes heavily depends upon synchronization of aggregation rounds. To enforce synchronization in these protocols, length of aggregation round should be long enough so that all the nodes in the system complete their aggregation information exchange. In this paper, we study the impacts of asynchrony in epidemic-style aggregation protocols. We present a simple asynchronous technique to estimate system aggregates in a distributed system. Based upon this technique, we analyze two popular existing epidemic-style aggregation protocols, Push-Pull and Push-Sum. Through detailed simulations, we evaluate accuracy and cost of asynchronous version of these protocols. We found that to obtain an estimate of the true system aggregate, aggregation protocols do not need to be synchronized and hence an efficient estimate can be obtained in lesser time.

Gossip-based protocols are a popular approach to estimate aggregates in P2P systems. For gossip-based aggregation protocols, the accuracy of the estimated aggregate heavily depends upon conservation of the system mass. In this paper, we present a gossip-based asynchronous aggregation protocol. Our protocol conserves the system mass under asynchronous settings and churn. Simulation results show that our proposed protocol converges to the correct system aggregate even in the presence of very dynamic and adverse network delays and under churn.

Fuzzy boundaries and unpredictable anomalies displayed in the raw sensor data make the process of defining a strong ellipsoid boundary for the raw data in the ellipsoid-based anomaly detection algorithms in wireless sensor networks a difficult problem. We have shown, using synthetic and real sensor data, that the random variable that represents the difference between any two randomly selected raw data points follows an identically independently distributed Gaussian distribution. We have used this statistical property to calculate ellipsoid boundaries for the Gaussian distribution which displays a robust ellipsoid shape and then to map each point of the distribution function to its corresponding raw data point to isolate anomalies from the sensor data. We have demonstrated the performance of the proposed approach by comparing it with the standard approach using both synthetic datasets and real Intel Berkeley Research Laboratory and Grand St Bernard datasets.

Due to the significant amount of energy consumed by chemical sensors for sensing, reducing sensing activity is critical for improving the lifespan of chemical sensor networks. In this paper, we consider a simple decentralized dynamic sensor activation protocol that aims to maintain a majority of sensors in the inactive (passive) state in the absence of a chemical attack, and rapidly activate the sensors when an attack is detected. This paper proposes two analytical models to study the behavior of the sensor network under the proposed protocol. Our first analytical model employs the known analogy between the information spread in a sensor network and the propagation of epidemics across a population. The second model describes the protocol by using a framework of graph theory.

Flooding-based Distributed Denial of Service (DDoS) attacks present a serious threat to the stability of the Internet. Identifying the attacks rapidly and accurately is significant for the efficient operation of Internet applications and services. Recent observations in the U.S. indicate a significant increase of cyber attacks on U.S. military information systems in 2009. Current technologies are still unable to withstand large-scale DDoS attacks. Single point detection and response is a first step to defeat such distributed attacks. Distributed global defense systems, using a coordinated effort, go much further towards thwarting such attacks. In this paper, we propose a distributed defense infrastructure to detect DDoS attacks globally using a cooperative overlay network and a gossip-based information exchange protocol. Our NS2 based simulation results show that the proposed solution can detect attacks with a detection rate as high as 0.99 with false alarms below 0.01. This compares favorably against other widely known methods including change-point detection, TTL analysis and wavelet analysis.

International Symposium on High Performance Distributed Computing (HPDC)

Ubiquitous network connectivity is challenged in areas outside of the range of wireless base stations and cellular networks. In such networks the mobile nodes are disconnected for significantly long periods of time. In this paper, we propose an approach where mobile nodes deposit/retrieve messages to/from known point locations in the geographic region. Messages are delivered from a source by being deposited at one or more locations that are later visited by the destination. We propose the use of passive, read/writable radio frequency identification tags, one per point location. Our simulation results indicate that our approach can achieve competitive message latency and delivery rates. We also provide techniques for optimizing the tag placement, including tag pruning, probability based tag distribution and a genetic algorithm; the genetic algorithm is shown to provide the best solutions to this problem.

The increasing number of stealthy and coordinated attacks on the Internet pose a significant threat to network security. Collaborative intrusion detection systems (CIDSs) have therefore been proposed to address this coordinated defense challenge by correlating patterns of suspicious activity based on the source addresses of the suspicious incoming traffic. However, during worm outbreaks, there can be a rapid growth in suspicious evidence that is reported about individual sources of the worm outbreak. In CIDSs that correlate suspicious activity by source address, the evidence relating to these worm spread sources can cause a load “hot-spot”, which severely degrades the overall performance of the detection system. In this paper, we propose a load balancing scheme for a CIDS to evenly distribute the workload to avoid hot-spots during worm outbreaks. Rather than correlating suspicious evidence based on source addresses, we distribute the load in the CIDS using a scheme that enables different possible patterns of suspicious evidence to be automatically mapped onto different processing nodes in the CIDS. Simulation results show that our scheme can achieve significant improvements in load balancing without sacrificing detection accuracy.

The decoupled nature of publish-subscribe (pub-sub) paradigm has gained popularity as a communication mechanism for distributed systems developed for decentralised networks. However, the lack of configurable and measurable properties in current decentralised pub-sub services introduces the issue of limited customisation. This shortcoming leads to the inability to meet different requirements needed by various application domains. In this paper, we address the challenge of supporting configurable properties in an overlay-oriented pub-sub scheme for decentralised networks. We use the measurable feature produced by the configurable properties to develop a mathematical model to analyse the behaviour of the pub-sub scheme with respect to important input factors such as distributions of subscriptions and events. These input factors ultimately affect the widely regarded performance requirement. Finally, we demonstrate the feasibility of our scheme and the validity of the derived model with a series of simulations.

The lack of preparedness of software engineering (SE) graduates for a professional career is a common complaint raised by industry practitioners. The career progression of many new graduates is severely impacted due to the lack of well rounded skills. For example, some of the technically stronger graduates lack communication and managerial skills and vise versa. Industry based capstone projects, incorporated as a part of an undergraduate degree, are a well accepted means of preparing students for their professional careers. Software Engineering undergraduates at the University of Melbourne engage in such industry based projects both in the penultimate and final years of their degree. Though aimed at providing students a real-life SE experience and preparing them for industry, we observed these projects to fail in some cases in giving the necessary breadth of skills. We believe this failure to be due to the lack of an objective framework to guide student learning outcomes during projects. To address this problem we developed an objective skill-based framework, focusing on managerial, engineering and personal skills. In this paper we present this framework and share our experiences of using it.

Efficient resource discovery mechanism is one of the fundamental requirement for grid computing systems, as it aids in resource management and scheduling of applications. Resource discovery involves searching for resources that match the user's application requirements. Various kinds of solutions to grid resource discovery have been developed, including the centralised and hierarchical information server approach. However, these approaches have serious limitations in regards to scalability, fault-tolerance and network congestion. To overcome such limitations, we propose a decentralised grid resource discovery system based on a spatial publish/subscribe index. It utilises a distributed hash table (DHT) routing substrate for delegation of d-dimensional service messages. Our approach has been validated using a simulated publish/subscribe index that assigns regions of a d-dimensional resource attribute space to the grid peers in the system. We generated the resource attribute distribution using the configurations obtained from the top 500 supercomputer list. The simulation study takes into account various parameters such as resource query rate, index load distribution, number of index messages generated, overlay routing hops and system size. Our results show that grid resource query rate directly affects the performance of the decentralised resource discovery system, and that at higher rates the queries can experience considerable latencies. Further, contrary to what one can expect, system size does not have a significant impact on the performance of the system, in particular the query latency.

Pervasive augmented reality gaming and mobile P2P are both attracting significant focus and much progress has occurred recently. Augmented reality games extend reality with virtual elements to enhance game experience and mobile P2P networks provide autonomous, self-organizing, scalable and robust communication platforms. We review the technological challenges in both mobile P2P computing and AR domains by investigating some existing related works. Then we propose possible directions of the mobile AR collaborative gaming over P2P networks and show its advantages over other AR gaming and discuss the feasibility and future requirements regarding the support for this type of game.

An important problem in network intrusion detection is how to detect large scale coordinated attacks such as scans, worms and denial-of-service attacks. These coordinated attacks can be difficult to detect at an early stage, since the evidence of the attack may be widely distributed across different subnetworks in the Internet. A critical issue for research is how to detect these large scale attacks by correlating information from multiple intrusion detection systems in an efficient manner. Several collaborative detection systems have been proposed in the literature. However, these proposals have lacked large scale testing in real networks, and the practicalities of how to optimize the trade-off between detection accuracy and reaction time of these systems has not been demonstrated. To address these challenges, we propose LarSID, a scalable decentralized large scale intrusion detection framework. LarSID provides a service for defending against attacks by sharing potential evidence of intrusions between participant intrusion detection systems via a distributed hash table (DHT) architecture. In particular, we investigate how to optimize the trade-off between detection accuracy and reaction time of LarSID based on an analysis of a large, real-world intrusion detection dataset (DShield Dataset), which has been collected from over 1600 firewall administrators across the world. LarSID has been deployed and tested on the PlanetLab testbed, and is built on top of OpenDHT - a public DHT service. Our experimental results show significant reductions in detection latency compared to a centralized detection architecture. Currently, LarSID has been deployed on 128 PlanetLab nodes as a large scale intrusion detection service.

Quality of service (QoS) is one of the important factors when choosing a Web service (WS) provided by a particular WS provider. In this paper we focus on performance, which is one of the important QoS attributes. Performance can be evaluated from the client-side as well as the server-side. As clients are connected through heterogeneous network environments, different clients can experience different performance although they are connected to the same WS provider. Through client grouping and profiling, performance experienced by prospective clients can be estimated based on other clients' historical performance. In this paper we present the common factors for client grouping and profiling, and present the results of the experiments we carried out to understand the effect of these factors on the performance experienced by WS clients. The experimental results show the importance of client grouping and profiling for WS recommendation and composition.

Agent technology is increasingly being used to develop software systems in different domains. Hence particular design problems are recurring in different multiagent systems development projects. In order to enhance the reuse of proven solutions to recurring design problems, agent design patterns are being identified and documented. Understandably, most of the available work on agent design patterns reflect object oriented concepts (classes, objects, inheritance, etc) and exhibit an implementation bias. It is preferable for these patterns to reflect the notions of agency and be described at the right level of abstraction. In this paper, we present a new agent design pattern classification scheme that is structured to better reflect the notions of agency and allow varying levels of abstraction in describing agent design patterns. We also show how four different agent design patterns fit into our classification scheme.

A visual model that gives a distortion measure for blocking artifacts in images is presented. Given the original and reproduced image as inputs, the model output is a numerical value that quantifies the visibility of blocking error in the reproduced image. The model is derived based on the human visual sensitivity to horizontal and vertical edge artifacts that result from blocking. Psychovisual experiments have been carried out to measure the visual sensitivity to these artifacts. In the experiments, typical edge artifacts are shown to subjects and the sensitivity to them is measured with the variation of background luminance, background activity, edge length, and edge amplitude. Synthetic test patterns are used as background images in the experiments. The sensitivity measures thus obtained are used to estimate the model parameters. The final model is tested on real images, and the results show that the error visibility predicted by the model correlates well with the subjective ranking.

A visual model which gives a distortion measure for coding artifacts in images is presented. Given the original and the reproduced image as inputs, the model output is a numerical value, which quantifies the visibility of error in a region of the reproduced image. The model aims to predict accurately the visibility of the common types of coding artifacts blocking, ringing and blurring. Visual sensitivity to these artifacts is measured using psyche-visual experiments. The sensitivity measures thus obtained are used to derive the visual model which predicts the visibility of error. The final model is tested on real images, and the results show that the error visibility predicted by the model correlates well with the subjective ranking.>

A visual model which gives a distortion measure for Blocking Artifacts in images is presented. Given the original and the reproduced image as inputs, the model output is a single numerical value which quantifies the visibility of blocking error in the reproduced image. The model is derived based on the human visual sensitivity to horizontal and vertical edge artifacts which result from blocking. Psychovisual experiments have been carried out using a novel experimental technique to measure the sensitivity to edge artifacts with the variation of edge length, edge amplitude, background luminance and background activity. The model parameters are estimated based on these sensitivity measures. The final model has been tested on real images, and the results show that the error predicted by the model correlate well with the subjective ranking.