The development of pervasive personal digital devices such as phones, watches, and headphones, interconnected by technologies such as Bluetooth, has led to a disruptive change in the ways in which local governments can monitor both vehicular and pedestrian traffic flows within their cities. In modern vehicles, navigation systems interconnect the personal devices of drivers and passengers typically via Bluetooth. By continuously monitoring such devices when they are in discover mode, traffic flows can be estimated almost in real-time. This paper examines traffic data collected from a Bluetooth Traffic Monitoring Systems installed by the Wellington City Council. Potentially such an installation could assist a local authority in real-time monitoring of normal traffic, as well as events including concerts and sport events, or in reaction to unanticipated events such as disasters. The limitation in this technology is that the data collected is of low fidelity, that is: not every vehicle has a detectable device, some have many, and there are devices carried by pedestrians and non-motor vehicles as well as stationary devices. This paper enumerates and investigates these challenges through statistical modelling, cleaning and data analysis. We present two novel algorithms for the processing of Bluetooth traffic data and validate our algorithms against a physical road counter. A case study is of a major earthquake is then presented as a proof of concept. The earthquake led to road closures, building collapse and other infrastructure damage and we examine three weeks of BTMS data and visualise how this earthquake impacted daily traffic flows.

In this paper we present a novel and comprehensive simulation framework that we have named AMEM (A Multidisciplinary Evacuation Model) for vehicle traffic modelling in urban areas – with a specific focus on large-scale evacuation scenarios. In general, the value of a comprehensive urban traffic modelling system is that it can assist authorities in identifying parts of a road transportation network that exhibit poor performance, or unanticipated and negative emergent properties under a variety of conditions. Such conditions can arise from planned or projected changes to the road infrastructure, or more interestingly, in reaction to uncommon or rare 100 year events. These are not the typical day to day traffic events that can be monitored and measured directly. In AMEM, we combine a number of different elements in our modelling, including routing, car-following, behaviour, driving culture, traffic light signalling, and psychological patterns. We validated the AMEM framework using real traffic data harvested from a network of Bluetooth and road sensors deployed in Wellington, New Zealand, and used this data as the basis for 13 scenarios. We also included in our study a unique socio-technical factor – the use of navigation systems, in part to address the question as to if such systems help or hinder traffic movement during an evacuation. Our results suggest that the use of navigation systems, as currently implemented, have a potentially negative impact on the evacuation process in dense urban areas.

In this paper we present DRIVE, a distributed service-based system designed to facilitate an open economic market for federating Cloud providers. To address the challenges associated with market ownership and operation we propose the use of a co-operative (co-op) infrastructure in which the services that make up DRIVE are hosted across participants' resources. To prevent malicious behavior we use cryptographic, secure and privacy preserving allocation protocols as a means of establishing trust in the allocation infrastructure. We investigate through simulation the effect of different strategies, pricing functions, and penalty models on allocation performance and revenue, and show that the overhead of running DRIVE's services on commodity infrastructure is modest.

Basic science is becoming ever more computationally intensive, increasing the need for large-scale compute and storage resources, be they within a High Performance Computer cluster, or more recently within the cloud. In most cases, large scale scientific computation is represented as a workflow for scheduling and runtime provisioning. Such scheduling become an even more challenging problem on cloud systems due to the dynamic nature of the cloud, in particular, the elasticity, the pricing models (both static and dynamic), the non-homogeneous resource types, the vast array of services, and virtualization. This mapping of workflow tasks on to a set of provisioned instances is an example of the general scheduling problem and is NP-complete. In addition, we also need to ensure that certain runtime constraints are met - the most typical being the cost of the computation and the time which that computation requires to complete. In this article, we introduce a new heuristic scheduling algorithm, Budget Deadline Aware Scheduling (BDAS), that addresses eScience workflow scheduling under budget and deadline constraints in Infrastructure as a Service (IaaS) clouds. The novelty of our work is satisfying both budget and deadline constraints while introducing a tunable cost-time trade off over heterogeneous instances. In addition, we study the stability and robustness of our algorithm by performing sensitivity analysis. The results demonstrate that overall BDAS finds a viable schedule for more than 40000 test cases accomplishing both defined constraints: budget and deadline. Moreover, our algorithm achieves a 17.0-23.8 percent higher success rate when compared to state of the art algorithms.

The Information and Communications Technology (ICT) industry in New Zealand is growing rapidly. The traditional university courses are preparing an insufficient number of graduates to sustain the growth. Many of the traditional graduate students lack key soft skills that are important in team based software development. This paper reports on the development of a conversion Master of Software Development degree. The students are all graduates with little or no computer science degrees, are taught key programming skills, with a focus on agile development. The programme begins by focusing on individual programming skills through solving problems. Later industrial partners are engaged by providing industrial problems to agile teams of students. The industrial partners are active partners in the agile teams as product owners. By solving the problems, the students develop both technical and non-technical skills while utilizing the skills obtained from previous studies. The results from the first year of the programme are encouraging. A key result is that a high number of students found work in paid IT positions before graduating. The main issue of the first year was introducing too many topics at the same time, over-assessment, not enough communication and contact time, little opportunity for the students to make their own experiences, and learning by making mistakes. The programme has been changed for the next year/s cohort to introduce less topics at once, provide time and space for learning, and a redesign of scheduling assessments.

Scientific collaboration networks are social networks in which vertices represent scientists and edges typically represent co-authorship. Such networks not only permit research into understanding the characteristics of scientific collaboration, but can also provide a basis for building collaborative research platforms to support research groups with functionality such as, information sharing, data repositories, attribution and communication. Collaboration networks are highly clustered, mapping closely to the real world relationships of individual researchers. However, just as eScience and big data constitute a well recognised disruptive change to the way basic research is carried out in many research fields, there is an equivalent and largely unexplored change in the collaborative relationships between researchers - which are becoming not only larger in scale, but also more distributed and interdisciplinary. One element in this, which we suggest will play a pivotal role in the future, is the formation of teams for large eScience and big data projects. This paper presents an innovative algorithm for expert team formation called Chemistry Oriented Team Formation (ChemoTF) based on two new metrics; Chemistry Level and Expertise Level. Chemistry Level measures scale of communication required by the task, while Expertise Level measures the overall expertise among potential teams filtered by Chemistry Level. This approach is tested using a large expertise corpus containing 472,365 individual authors. The ChemoTF algorithm is able to build teams for median average 90% of the expected cost, achieving 99% fit while remaining tractable for teams up to 16 individuals - resulting in the formation of more communicative and cost effective teams with higher expertise level.

In this paper we present a new, very large, rich, Comprehensive Scholarly Corpus (CompScholarCorp) as a platform and data source for future research. Our corpus contains records of 1,044,454 papers, 472,365 unique authors, and substantial publication meta-data for each record. We have integrated the data we collected from 276 publishers using a uniform and consistent XML data format within the corpus. The corpus is designed to be compatible with DBLP enabling existing research to utilise our new corpus directly. As an initial analysis of the corpus, we present a number of visualisations of the corpus to better understand the data, provide some analytics of the data, and present a rule-of-thumb we have observed for citations.

Cloud providers offer a diverse set of instance types with varying resource capacities, designed to meet the needs of a broad range of user requirements. While this flexibility is a major benefit of the cloud computing model, it also creates challenges when selecting the most suitable instance type for a given application. Sub-optimal instance selection can result in poor performance and/or increased cost, with significant impacts when applications are executed repeatedly. Yet selecting an optimal instance type is challenging, as each instance type can be configured differently, application performance is dependent on input data and configuration, and instance types and applications are frequently updated. We present a service that supports automatic profiling of application performance on different instance types to create rich application profiles that can be used for comparison, provisioning, and scheduling. This service can dynamically provision cloud instances, automatically deploy and contextualize applications, transfer input datasets, monitor execution performance, and create a composite profile with fine grained resource usage information. We use real usage data from four production genomics gateways and estimate the use of profiles in autonomic provisioning systems can decrease execution time by up to 15.7% and cost by up to 86.6%.

Scientific research is increasingly reliant on big compute and big data, the fusion of which is known as data intensive science. Large scale scientific analyses are typically represented as workflows which are the typical model for characterizing e-science experiments in distributed systems. Workflows with a large number of tasks are distributed in parallel across computing resources to speed up analyses. The provision of compute capabilities is undergoing a rapid migration from dedicated infrastructure to the cloud. This migration is fuelled by dynamic infrastructure scalability with changes in demand. Cloud instances incur different costs and execution time with different configurations. A key concern for workflow scheduling is to make an appropriate trade-off between these two factors. In this paper, we introduce the Budget Distribution with Trickling (BDT) algorithm that presents new notions for distributing budget based on the dependency structure inherent in workflows. In addition we propose several new strategies for sharing or distributing the budget, and propose trickling to redistribute unspent budget down to other levels. Our results show that biasing the budget distribution to the earlier computation within a workflow will generally produce a lower makespan within budget.

An opportunity arose to secure a share of a multimillion dollar initiative to train additional ICT Postgraduate students. However, local and national ICT industries did not want such students, with the greatest demand being for conventional 4 year Bachelor of Engineering or 3 year Bachelor of Science graduates. This paper outlines a strategy that developed a suite of programmes that passed nine quality assurance stages and won a competitive bid to be selected as one of three preferred suppliers. Further, the programmes so developed addressed industry needs, and were able to target both domestic and international students.

Cloud computing provides an efficient model to host and scale scientific applications. While cloud-based approaches can reduce costs as users pay only for the resources used, it is often challenging to scale execution both efficiently and cost-effectively. We describe here a cost-aware elastic cloud provisioner designed to elastically provision cloud infrastructure to execute analyses cost-effectively. The provisioner considers real-time spot instance prices across availability zones, leverages application profiles to optimize instance type selection, over-provisions resources to alleviate bottlenecks caused by oversubscribed instance types, and is capable of reverting to on-demand instances when spot prices exceed thresholds. We evaluate the usage of our cost-aware provisioner using four production scientific gateways and show that it can produce cost savings of up to 97.2% when compared to naive provisioning approaches.

Cloud computing is often suggested as a low-cost and scalable model for executing and scaling scientific analyses. However, while the benefits of cloud computing are frequently touted, there are inherent technical challenges associated with scaling execution efficiently and cost-effectively. We describe here a cost-aware elastic provisioner designed to dynamically and cost-effectively provision cloud infrastructure based on the requirements of user-submitted scientific workflows. Our provisioner is used in the Globus Galaxies platform -- a Software-as-a-Service provider of scientific analysis capabilities using commercial cloud infrastructure. Using workloads from production usage of this platform we investigate the performance of our provisioner in terms of cost, spot instance termination rate, and execution time. We demonstrate cost savings across six production gateways of up to 95% and 12% improvement in total execution time when compared to a worst case scenario using a single instance type in a single availability zone.

Commercial clouds have increasingly become a viable platform for hosting scientific analyses and computation due to their elasticity, recent introduction of specialist hardware, and pay-as-you-go cost model. This computing paradigm therefore presents a low capital and low barrier alternative to operating dedicated eScience infrastructure. Indeed, commercial clouds now enable universal access to capabilities previously available to only large well funded research groups. While the potential benefits of cloud computing are clear, there are still significant technical hurdles associated with obtaining the best execution efficiency whilst trading off cost. Large scale scientific analyses are typically represented as workflows, in order to manage multiple tools and data sets. Mapping workflow tasks on to a set of provisioned instances is an example of the general scheduling problem and is NP-complete. In this case, the mapping includes elasticity, where as part of the mapping process additional instances may be provisioned. In this paper we present anew algorithm, Proportional Deadline Constrained (PDC), that addresses eScience workflow scheduling in the cloud. PDC's aim is to minimize costs while meeting deadline constraints. To validate the PDC algorithm, we constructed a Cloud Sim test bed and compared PDC with two other similar algorithms over three workflows. Our results demonstrate that overall PDC achieves generally lower costs for a given deadline, but more significantly, is usually able to construct a viable schedule with tight deadlines where the other algorithms studied cannot.

Commercial cloud providers are increasingly offering high performance and GPU-enabled resources capable of facilitating e-Science applications. However, the limitations of a public cloud's internal network performance are well documented and can lead to the decision to use dedicated infrastructure over cloud resources for scientific applications. This paper explores the potential for improvement in the performance of e-Science applications on public clouds through the examination of the network in more detail. We introduce health indicators and evaluate various tomographic techniques for their ability to infer information regarding the network connection between instances. We also propose and formulate a set of health markers and health metrics to efficiently assess the network over time in order to make informed deployment decisions. Finally, we evaluate our work through a real-world medical image reconstruction application.

Social network platforms have rapidly changed the way that people communicate and interact. They have enabled the establishment of, and participation in, digital communities as well as the representation, documentation and exploration of social relationships. We believe that as `apps' become more sophisticated, it will become easier for users to share their own services, resources and data via social networks. To substantiate this, we present a social compute cloud where the provisioning of cloud infrastructure occurs through “friend” relationships. In a social compute cloud, resource owners offer virtualized containers on their personal computer(s) or smart device(s) to their social network. However, as users may have complex preference structures concerning with whom they do or do not wish to share their resources, we investigate, via simulation, how resources can be effectively allocated within a social community offering resources on a best effort basis. In the assessment of social resource allocation, we consider welfare, allocation fairness, and algorithmic runtime. The key findings of this work illustrate how social networks can be leveraged in the construction of cloud computing infrastructures and how resources can be allocated in the presence of user sharing preferences.

Collaborative systems require access control to prevent unauthorised access and change. Access control has a number of issues, including administration and maintenance overheads. In this paper we argue that it is time to reconsider how access controls work, particularly with scientific and data related domains, and to this end we propose a new paradigm based on a user's demographics and behaviour, rather than simply their identity. In essence, it is both who you are and what you do that is important. We introduce Graft, our Generalised Recommendation Architecture that allows us to support a range of different recommendation models, and provide case studies to illustrate the usefulness of our architecture.

Social clouds are a relatively new paradigm that allow users of an underlying social network to share their resources with their "friends", using previously established relationships. However, this sharing has a number of issues, including granularity of friendships, resource costs and maintenance. In this paper we argue that sharing decisions should be based on relationship information augmented by supplementary metadata derived from multiple sources. Users should be able to leverage the information available on their non-uniform friend relationships when making decisions, allowing them to confidently share their resources with those that would normally be outside of their immediate social circle. We introduce Graft, our Generalised Recommendation Architecture, that provides us with a mechanism to support this new approach.

Utility computing models have long been the focus of academic research, and with the recent success of commercial cloud providers, computation and storage is finally being realized as the fifth utility. Computational economies are often proposed as an efficient means of resource allocation, however adoption has been limited due to a lack of performance and high overheads. In this paper, we address the performance limitations of existing economic allocation models by defining strategies to reduce the failure and reallocation rate, increase occupancy and thereby increase the obtainable utilization of the system. The high-performance resource utilization strategies presented can be used by market participants without requiring dramatic changes to the allocation protocol. The strategies considered include overbooking, advanced reservation, just-in-time bidding, and using substitute providers for service delivery. The proposed strategies have been implemented in a distributed metascheduler and evaluated with respect to Grid and cloud deployments. Several diverse synthetic workloads have been used to quantity both the performance benefits and economic implications of these strategies.

Volunteer computing provides an alternative computing paradigm for establishing the resources required to support large scale scientific computing. The model is particularly well suited for projects that have high popularity and little available computing infrastructure. The premise of volunteer computing platforms is the contribution of computing resources by individuals for little to no gain. It is therefore difficult to attract and retain contributors to projects. The Social Cloud for Volunteer Computing aims to exploit social engineering principles and the ubiquity of social networks to increase the outreach of volunteer computing, by providing an integrated volunteer computing application and creating gamification algorithms based on social principles to encourage contribution. In this paper we present the development of a production SoCVC, detailing the architecture, implementation and performance of the SoCVC Facebook application and show that the approach proposed could have a high impact on volunteer computing projects.

Online relationships in social networks are often based on real world relationships and can therefore be used to infer a level of trust between users. We propose leveraging these relationships to form a dynamic "Social Cloud,” thereby enabling users to share heterogeneous resources within the context of a social network. In addition, the inherent socially corrective mechanisms (incentives, disincentives) can be used to enable a cloud-based framework for long term sharing with lower privacy concerns and security overheads than are present in traditional cloud environments. Due to the unique nature of the Social Cloud, a social market place is proposed as a means of regulating sharing. The social market is novel, as it uses both social and economic protocols to facilitate trading. This paper defines Social Cloud computing, outlining various aspects of Social Clouds, and demonstrates the approach using a social storage cloud implementation in Facebook.

Scientific researchers faced with extremely large computations or the requirement of storing vast quantities of data have come to rely on distributed computational models like cloud computing. However, distributed computation is typically complex and expensive. The Social Cloud for Public eResearch aims to provide researchers with a platform to exploit social networks to reach out to users who would otherwise be unlikely to donate computational time for scientific and other research oriented projects. In this paper we explore the motivations of users to contribute computational time and examine the various ways these motivations can be catered to through established social networks. We specifically look at integrating Face book and BOINC, and discuss the architecture of the functional system and the novel social engineering algorithms that power it.

Social networks provide a useful basis for enabling collaboration among groups of individuals. This is applicable not only to social communities but also to the scientific community. Already scientists are leveraging social networking concepts in projects to form groups, share information and communicate with their peers. For scientific projects which require large computing resources, one useful aspect of collaboration is the sharing of computing resources among project members. A social network provides an ideal platform to share these resources. This paper introduces a framework for Social Cloud computing with a view towards collaboration and resource sharing within a scientific community. The architecture of a Social Cloud, where individuals or institutions contribute the capacity of their computing resources by means of Virtual Machines leased through the social network, is outlined. Members of the Social Cloud can contribute, request, and use Virtual Machines from other members, as well as form Virtual Organizations among groups of members.

With the increasingly ubiquitous nature of Social networks and Cloud computing, users are starting to explore new ways to interact with, and exploit these developing paradigms. Social networks are used to reflect real world relationships that allow users to share information and form connections between one another, essentially creating dynamic Virtual Organizations. We propose leveraging the pre-established trust formed through friend relationships within a Social network to form a dynamic “Social Cloud”, enabling friends to share resources within the context of a Social network. We believe that combining trust relationships with suitable incentive mechanisms (through financial payments or bartering) could provide much more sustainable resource sharing mechanisms. This paper outlines our vision of, and experiences with, creating a Social Storage Cloud, looking specifically at possible market mechanisms that could be used to create a dynamic Cloud infrastructure in a Social network environment.

Economic models have long been advocated as a means of efficient resource allocation, however they are often criticized due to a lack of performance and high overheads. The widespread adoption of utility computing models as seen in commercial Cloud providers has re-motivated the need for economic allocation mechanisms. The aim of this work is to address some of the performance limitations of existing economic allocation models, by reducing the failure/reallocation rate, increasing occupancy and thereby increasing the obtainable utilization of the system. This paper is a study of high performance resource utilization strategies that can be employed in Grid and Cloud systems. In particular we have implemented and quantified the results for strategies including overbooking, advanced reservation, justin-time bidding and using substitute providers for service delivery. These strategies are analyzed in a meta-scheduling context using synthetic workloads derived from a production Grid trace to quantify the performance benefits obtained.

In this paper we present DRIVE, a novel architecture for a virtual organization (VO) based distributed economic meta-scheduler in which members of the VO collaboratively allocate grid resources. Resource providers joining the VO contribute obligation services to the VO. These contributed services are in effect membership 'dues' and are used in the running of the VO's operations - allocation, advertising, management, etc. We use an auction plug-in mechanism to support arbitrary auction protocols which allows users to choose a protocol based on specific requirements and infrastructural availability. For instance, within a single organization, where internal trust exists, users can achieve maximum allocation performance by choosing a conventional sealed bid auction plug-in. In a global utility Grid no such trust exists. The same meta-scheduler architecture can be used with a more expensive secure auction protocol plug- in which ensures the allocation is carried out fairly in the absence of trust The DRIVE prototype has been implemented as a collection of standard WSRF Web services and includes a distributed implementation of a secure combinatorial Garbled Circuit protocol.

In this paper we present an autonomic Web services architecture that manages both the performance of service containers and the interconnection of those containers into a service overlay network. The advantages of this approach include the easing of management tasks through the autonomic systems ability to self-configure, self-optimise and self-heal. We also benefit from improved resilience and anticipate an improvement in overall performance. In our architecture we incorporate a structured distributed hash table peer-to-peer overlay network within our autonomic Web services container. Our architecture is inherently non-hierarchical, widely distributed and enables SLA compliant deployment of WSRF services. We have simplified the management of such a system by adhering to autonomic principles, and we maintain the performance of the system by tightly integrating SLA compliance and migrating services between containers to preserve QoS. We have developed a workable system for both service deployment and migration without the need for global state.

Service-oriented architectures provide integration of interoperability for independent and loosely coupled services. Web services and the associated new standards such as WSRF are frequently used to realise such service-oriented architectures. In such systems, autonomic principles of self-configuration, self-optimisation, self-healing and self- adapting are desirable to ease management and improve robustness. In this paper we focus on the extension of the self management and autonomic behaviour of a WSRF container connected by a structured P2P overlay network to monitor and rectify its QoS to satisfy its SIAs. The SLA plays an important role during two distinct phases in the life-cycle of a WSRF container. Firstly during service deployment when services are assigned to containers in such a way as to minimise the threat of SLA violations, and secondly during maintenance when violations are detected and services are migrated to other containers to preserve QoS. In addition, as the architecture has been designed and built using standardised modern technologies and with high levels of transparency, conventional Web services can be deployed with the addition of a SLA specification.

Commercialisation or globalisation of large scale grids requires the provision of mechanisms to share the wide pool of grid brokered resources such as computers, software, licences and peripherals amongst many users and organisations. Quickly and efficiently servicing resource requests is critical to the efficiency of such grid based utility computing and communication providers. The CORA architecture is a market based resource reservation system that utilises a trustworthy Vickrey auction to make combinatorial allocations of resources. The primary advantage of such a scheme is that a trusted auctioneer is no longer necessary, and any system entity can safely host a trustworthy auction. This approach results in more flexibility in the design of large economic systems, with the potential for wide distribution of load amongst many auctioneers. In addition, only the winners of the auction and the prices they pay are revealed while all other bid values are kept secret. This paper also provides performance results for our implementation, that identify the constraints within which a practical trustworthy auction scheme can be implemented in a grid-style economy.

The CORA (Coallocative, Oversubscribing Resource Allocation) architecture is a market based resource reservation system that utilises a trustworthy Vickrey auction to make combinatorial allocations of resources. This paper provides an overview of several significant components of the CORA architecture. Firstly, CORA utilises a novel combination of techniques to improve utilisation, including oversubscription, coallocation, just-in-time reallocation and a flexible contract structure. Secondly, this paper utilises a new auction architecture that does not require the auctioneers to be trusted. The advantage is that any entity (untrusted or otherwise) can conduct a privacy preserving Vickrey auction, removing the need for a trusted and privileged auction service within the system. CORA demonstrates how a practical, efficient and trustworthy auction scheme can be implemented in a Grid Economy.

The CORA (Coallocative, Oversubscribing Resource Allocation) architecture is an auction based resource reservation system that makes combinatorial allocations of resources to clients. The focus of this paper is on the use of cryptographic tools in CORA to remove the need for trust in the resource auctioneer. One of the nice properties of this approach is that the auctioneers can be drawn from an arbitrary pool of untrusted peers, without the need to establish pre-existing trust or restrict the role of auctioneer to a trusted system service. This approach results in more flexibility in the design of large economic systems, with the potential for wide distribution of load amongst many auctioneers. In addition, only the winners of the auction and the prices they pay are revealed while all other bid values are kept secret. It is our belief that future growth or commercialisation of large scale Grid systems requires the provision of such mechanisms to share the wide pool of Grid brokered resources such as computers, software, licences and peripherals amongst many users and organisations. This paper encapsulates an overview of our design, our experiences of implementing two different secure auction protocols and the performances that we have achieved.

In the effort to reach beyond 3G, researchers have been actively looking at utilizing new models for network based services. Small mobile, pervasive and ubiquitous devices will benefit from networked services and computation provided by utility computing providers and the virtual organizations that lease resources from them. As an additional factor, we believe that it is critical that the mobile, pervasive or ubiquitous devices be able to dynamically manipulate their resource specifications when obtaining services and resources from the utility computing and communication network. This requires a simple, manipulatable, and preferably modular resource specification structure. This paper presents the Resource Description Graph (RDG). The RDG is used to represent available and required resources for hosts and applications in a directed acyclic graph. The RDG has many desirable properties including inherent security, expressiveness, modularity, and composition. We show that the computational time to match RDG resource specifications, thirty resource types and constraints, is less than 1ms --- demonstrating that the RDG is a practical approach to resource specification with a low computational overhead.