While Software Engineering practice in the industry had matured well over the second half of the twentieth century, the first two decades of this millennium have witnessed disruptive growth of the field arising out of the evolution of new problem domains (e.g., intelligent applications and data management) as well as diverse deployment patterns (e.g., cloud, mobile and embedded devices). More subtle and impactful for practitioners is the change in perspective and approach-from a project-driven lifecycle to a product-driven lifecycle of software. Academic programs have kept up with the former set of changes as evidenced in curricular aspects or components such as DevOps, Mobile Application Development, Cloud-Native Application Development, Data Engineering, Big Data Systems, Software Engineering for AI, etc. But there is a distinct lack of representation of the product perspective or approach and the emerging role of Product Management in typical University-level Software Engineering programs and curricula. We propose to address this gap by identifying-and then incorporating into a curriculum-the following:(i)requirements and features of a Software Product Life Cycle,(ii)practices of a product-driven approach to Software Engineering, and(iii)competencies of a Software Product Manager.We show how these aspects can be addressed by a graduate level Software Engineering program with an optional specialization in Software Product Management targeted at practicing Software Engineers. We present the design of such a program in sufficient detail along with validation from industry and comparison with other programs.

Clustering of data streams has become very popular in recent times, owing to rapid rise of real-time streaming utilities that produce large amounts of data at varying inter-arrival rates. We propose AnyClus, a framework for anytime clustering of data streams. AnyClus uses a proposed variant of R-tree, AnyRTree, to capture the incoming stream objects arriving at variable rate, and to index them in the form of micro-clusters of hierarchical fashion. The leaf-level micro-clusters produced are aggregated and stored in a logarithmic tilted-time window framework (TTWF). Our extensive experimental analysis shows (i) the capability of AnyClus in handling variable stream speeds (upto 250k objects/second); (ii) its ability to produce micro-clusters of high purity (≈1) and compactness; (iii) effectiveness of AnyRTree in handling noise, capturing concept drift and preservation of spatial locality in the indexing of micro-clusters, when compared to the existing methods. We also propose a parallel framework, Any-MP-Clus, for anytime clustering of multiport data streams over commodity clusters. Any-MP-Clus uses AnyRTree at each computing node of the cluster (for each stream-port) and maintains the aggregated micro-clusters in TTWF. The experimental results on datasets of billions scale show that Any-MP-Clus is scalable, efficient and produces clustering of higher quality.

Hierarchical Agglomerative Clustering (HAC) algorithms are used in many applications where clusters have a hierarchical relationship between them. Their parallelization is challenging due to the dependence of every agglomeration step on all previous agglomerations. Although a few parallel algorithms have been proposed for SLINK HAC algorithm, only limited work has been done to parallelize other HAC algorithms. In this paper, we present a high-level abstraction, which provides a uniform way to specify any HAC algorithm, and a framework for automatic parallelization of the same for distributed memory systems. The abstraction is supported by constructs in a high level, domain specific language, and a compiler translates algorithms expressed in this language to efficient parallel code targeting distributed systems. Our experiments on multiple HAC algorithms proves that the runtime performance achieved is comparable with state-of-the-art manual parallel implementations on Spark and MPI while requiring only a fraction of the programming effort. At runtime, master-slave execution is used, and load is balanced among the slaves in an algorithm-agnostic way, which is a significant contrast to custom load-balancing techniques seen in the literature on parallel HAC algorithms.

Big Data has significantly increased the dependence of data analytics community on High Performance Computing (HPC) systems. However, efficiently programming an HPC system is still a tedious task requiring specialized skills in parallelization and the use of platform-specific languages as well as mechanisms. We present a framework for quickly prototyping new/existing density-based clustering algorithms while obtaining low running times and high speedups via automatic parallelization. The user is required only to specify the sequential algorithm in a Domain Specific Language (DSL) for clustering at a very high level of abstraction. The parallelizing compiler for the DSL does the rest to leverage distributed systems - in particular, typical scale-out clusters made of commodity hardware. Our approach is based on recurring, parallelizable programming patterns known as Kernels, which are identified and parallelized by the compiler. We demonstrate the ease of programming and scalable performance for DBSCAN, SNN, and RECOME algorithms. We also establish that the proposed approach can achieve performance comparable to state-of-the-art manually parallelized implementations while requiring minimal programming effort that is several orders of magnitude smaller than those required on other parallel platforms like MPI/Spark.

Ease of programming and optimal parallel performance have historically been on the opposite side of a tradeoff, forcing the user to choose. With the advent of the Big Data era and rapid evolution of sequential algorithms, the data analytics community can no longer afford the tradeoff. We observed that several clustering algorithms often share common traits - particularly, algorithms belonging to same class of clustering exhibit significant overlap in processing steps. Here, we present our observation on domain patterns in Representative-based clustering algorithms and how they manifest as clearly identifiable programming patterns when mapped to a Domain Specific Language (DSL). We have integrated the signatures of these patterns in the DSL compiler for parallelism identification and automatic parallel code generation. Our experiments on different state-of-the-art parallelization frameworks shows that our system is able to achieve near-optimal speedup while requiring a fraction of the programming effort, making it an ideal choice for the data analytics community.

Parallelizing data mining algorithms has become a necessity as we try to mine ever increasing volumes of data. Spatial data mining algorithms like Dbscan, Optics, Slink, etc. have been parallelized to exploit a cluster infrastructure. The efficiency achieved by existing algorithms can be attributed to spatial locality preservation using spatial indexing structures like k-d-tree, quad-tree, grid files, etc. for distributing data among cluster nodes. However, these indexing structures are static in nature, i.e., they need to scan the entire dataset to determine the partitioning coordinates. This results in high data distribution cost when the data size is large. In this paper, we propose a dynamic distributed data structure, DD-Rtree, which preserves spatial locality while distributing data across compute nodes in a shared nothing environment. Moreover, DD-Rtree is dynamic, i.e., it can be constructed incrementally making it useful for handling big data. We compare the quality of data distribution achieved by DD-Rtree with one of the recent distributed indexing structure, SD-Rtree. We also compare the efficiency of queries supported by these indexing structures along with the overall efficiency of DBSCAN algorithm. Our experimental results show that DD-Rtree achieves better data distribution and thereby resulting in improved overall efficiency.

Single linkage (SLINK) hierarchical clustering algorithm is a preferred clustering algorithm over traditional partitioning-based clustering as it does not require the number of clusters as input. But, due to its high time complexity and inherent data dependencies, it does not scale well for large datasets. In this paper, we parallelize an efficient implementation of SLINK algorithm to leverage a commodity cluster of multicore workstations. We present, dGridSlink, a distributed algorithm, which outperforms the best existing parallel solution in literature for all the real datasets considered. We also propose a hybrid parallel algorithm hGridSLINK for a cluster of multicore nodes. The proposed parallel algorithms are scalable and can cluster (several) millions of data points efficiently, without compromising the quality of clustering.

Clustering is a popular data mining and machine learning technique which discovers interesting patterns from unlabeled data by grouping similar objects together. Clustering high-dimensional data is a challenging task as points in high dimensional space are nearly equidistant from each other, rendering commonly used similarity measures ineffective. Subspace clustering has emerged as a possible solution to the problem of clustering high-dimensional data. In subspace clustering, we try to find clusters in different subspaces within a dataset. Many subspace clustering algorithms have been proposed in the last two decades to find clusters in multiple overlapping subspaces of high-dimensional data. Subspace clustering algorithms iteratively find the best subset of dimensions for a cluster from 2d-1 possible combinations in d-dimensional data. Subspace clustering is extremely compute intensive because of exhaustive search of subspaces, especially in the bottom-up subspace clustering algorithms. To address this issue, an efficient parallel framework for grid-based bottom-up subspace clustering algorithms is developed, considering popular algorithms belonging to this category. The framework is implemented for shared memory, distributed memory, and hybrid systems and is tested for three grid-based bottom-up subspace clustering algorithms: CLIQUE, MAFIA, and ENCLUS. All parallel implementations exhibit impressive speedup and scalability on real datasets.

Clustering is a popular data mining technique which discovers structure in unlabeled data by grouping objects together on the basis of a similarity criterion. Traditional similarity measures lose their meaning as the number of dimensions increases and as a consequence, distance or density based clustering algorithms become less meaningful. Shared Nearest Neighbor (SNN) is a solution to clustering high-dimensional data with the ability to find clusters of varying density. SNN assigns objects to a cluster, which share a large number of their nearest neighbors. However, SNN is compute and memory intensive for data of large size and/or dimensionality. Nearest neighbor queries are responsible for a major proportion of computations in SNN, resulting in lower efficiency for higher value of number of nearest neighbors (k). The main motivation of this work is to improve the efficiency of SNN and to parallelize it so that it can be used for clustering large high-dimensional datasets and for large values of k. Existing SNN algorithms become inefficient in these situations. In this paper, we present a new sequential SNN algorithm, R-SNN, which uses R-tree for executing neighborhood queries efficiently and exploiting spatial locality to minimize memory usage. R-SNN is benchmarked against the best available implementation of SNN and is found up to 77 times faster when tested on various real datasets. R-SNN is parallelized for distributed memory, shared memory, and hybrid systems. Significant speedup and scalability achieved can be attributed to parallelization and good load balancing strategies and also to exploitation of spatial locality. Experimental results demonstrate the same for datasets of varying dimensionality and size. The maximum speedup achieved for shared, distributed, and hybrid models are 427.19 using 48 threads, 394.24 using 32 processes, and 1380.69 on 32 nodes (with each node spawning 4 threads), respectively. Super-linear speedup for some datasets is attributed to optimized neighborhood queries. All the proposed algorithms produce identical clustering results as that of the classical SNN.

Web-business platforms offer business capabilities as enterprise services hosted in a multi-tenant cloud environment. Often there is a need to create heavyweight variants of these enterprise services to support: inherent variability in the underlying business process, industry-specific requirements, globalization concerns and customer-specific requirements. These variability concerns affect both the service interface and the service provider implementation and hence are crosscutting in nature. In this paper, we use principles of aspect-oriented software development to modularize these variability concerns. We also provide an aspect specification scheme to specify these concerns. We propose an approach to create heavyweight service variants centered on a service kernel, which forms a common service core across tenants. Heavyweight service variants are created by weaving aspects into the service kernel. Our approach provides improved governance for the provider while offering maximum flexibility for the consumers.

Nloblle computing devices intended for disconnected operation, such as laptops and personal organizers, must employ optimistic replication strategi~for user files.Unlike traditional distributed systems.such devices do not attempt to present a "single filesystem" semanti~users are aware that their fles are replicated, and that updates to one rephca till not be seen in another until some point of synchronization is reached (often under the user's exphcit control).A variety of tools, collectively called file synchronizers, support this mode of operation.Unfortunately, present-day synchronizers seldom give the user enough information to predict how they will behave under all circumstances.Simple slogans fike "Non-confecting updates are propagated to other replicas" ignore numerous subtletim-e.g., Precisely what constitutes a confict be @een updates in different replicas?What does the synchronizer do if updatw confict?What happens when fles are renamed?What if the directory structure is reorganized in one replica?Our god is to offer a simple, concrete, and precise frame work for describing the behavior of file synchronizers.To this end, n?edivide the synchronization task into two conceptually distinct phasm update detection and Reconciliation.We dEcuss each phase in detail and develop a straightforn'ard specification of each.We sketch our on prototype implementation of these specifications and discuss how they apply to some existing synchronization tools. IntroductionThe grotih of mobile computing has brought to fore novel issues in data management, in particular data reification under disconnected operation.Support for rephcation can be provided either transparently (tith flesystem or database support for cfient-side caching, transaction logs, etc.) or by user-visible tools for exThcit rephca management.In this paper we investigate one class of user-visible tools-commonly called file syrtchTonizeTs-w"hich allow, updates in different repficas to be reconciled at the user's request.