The past decade has seen a tremendous interest in large-scale data processing at Microsoft. Typical scenarios include building business-critical pipelines such as advertiser feedback loop, index builder, and relevance/ranking algorithms for Bing; analyzing user experience telemetry for Office, Windows or Xbox; and gathering recommendations for products like Windows and Xbox. To address these needs a first-party big data analytics platform, referred to as Cosmos, was developed in the early 2010s at Microsoft. Cosmos makes it possible to store data at exabyte scale and process in a serverless form factor, with SCOPE [4] being the query processing workhorse. Over time, however, several newer challenges have emerged, requiring major technical innovations in Cosmos to meet these newer demands. In this abstract, we describe three such challenges from the query processing viewpoint, and our approaches to handling them.

The increasing size of software systems and the emergence of increasingly autonomous systems demands innovative software engineering practices. The way software is being developed and maintained is rapidly changing and must take into account multifaceted constraints like fast-changing and unpredictable markets, complex and changing customer requirements, pressures of shorter time-to-market, and rapidly advancing information technologies, to name just a few of these new aspects. To cope with such constraints, software is increasingly produced according to rapid continuous software engineering development processes. The International Conference on Software Architecture (ICSA), is the premier platform for academia and industry to join efforts in addressing these challenges, bringing innovative solutions to the problems we face in the software architecture domain.

Enabling interactive data exploration at cloud scale requires minimizing end-to-end query execution latency, while guaranteeing fault tolerance, and query execution under resource-constraints. Typically, such a query execution involves orchestrating the execution of hundreds or thousands of related tasks on cloud scale clusters. Without any resource constraints, all query tasks can be scheduled to execute simultaneously (gang scheduling) while connected tasks stream data between them. When the data size referenced by a query increases, gang scheduling may be resource-wasteful or un-satisfiable with a limited, per-query resource budget. This paper introduces B ubble E xecution , a new query processing framework for interactive workloads at cloud scale, that balances cost-based query optimization, fault tolerance, optimal resource management, and execution orchestration. Bubble execution involves dividing a query execution graph into a collection of query sub-graphs (bubbles), and scheduling them within a per-query resource budget. The query operators (tasks) inside a bubble stream data between them while fault tolerance is handled by persisting temporary results at bubble boundaries. Our implementation enhances our JetScope service, for interactive workloads, deployed in production clusters at Microsoft. Experiments with TPC-H queries show that bubble execution can reduce resource usage significantly in the presence of failures while maintaining performance competitive with gang execution.

Software engineering is foremost an engineering discipline. Engineering in general and software engineering specifically has to balance many factors to achieve viable tradeoffs - an understanding of the factors as well as the viability criteria is at the heart of the educational challenge. All engineering has one ultimate goal: the delivery of artifacts (products, commercial or not) that meet the needs of those using such artifacts. All engineering lives in the intersection of people, technology, domain, and opportunity aspects. Software engineering, however, is laden with its own specific difficulties. Software as an engineering medium fills a space between the fluidity of digital content, with which software shares the representation, and the nature of machines, with which software shares the flexible and repeatable application.

Software components, if used properly, offer many software engineering benefits. Yet, they also pose many original challenges starting from quality assurance and ranging to architectural embedding and composability. In addition, the recent movement towards services, as well as the established world of objects, causes many to wonder what purpose components might have. This extended abstract summarizes the main points of my Frontiers of Software Practice (FOSP) talk at ICSE 2003. The topics covered aim to offer an end-to-end overview of what role components should play, where they should be used, and how this can be achieved Some key open problems are also pointed out.

Web services are emerging technologies to reuse software as services over the Internet by wrapping underlying computing models with XML. Web services are rapidly evolving and are expected to change the paradigms of both software development and use. This panel will discuss the current status and challenges of Web services technologies.

Sudden and significant demand for B2B process automation has seen the entry of workflow management systems (WFMS) into the component arena. Workflows offer highly expressive and graphical process control constructs for the coordinative component assembly, however, current provisions seem more suitable for internal process pipelines in single organizations, built without future reuse in mind. In this paper, we identify particular areas of workflow legacy which obstruct flexible reuse and composition under B2B assembly. New abstractions are identified for the tighter multi-lateral coupling of workflows such that: synchronization is possible across encapsulated workflows boundaries and external interactions occur through blackbox interfaces. Against the stifled efforts of loosely-coupled WFMS interoperability, a top-down architectural strategy is charted, where regulation can occur "above" workflow components - at a higher-tier workflow component framework accepting heterogeneous WFMSs as "plug-ins".

Sather extends the notion of an iterator in a powerful new way. We argue that iteration abstractions belong in class interfaces on an equal footing with routines. Sather iterators were derived from CLU iterators but are much more flexible and better suited for object-oriented programming. We retain the property that iterators are structured , i.e., strictly bound to a controlling structured statement. We motivate and describe the construct along with several simple examples. We compare it with iteration based on CLU iterators, cursors, riders, streams, series, generators, coroutines, blocks, closures, and lambda expressions. Finally, we describe experiences with iterators in the Sather compiler and libraries.