In this paper we describe a visualization system that shows the behavior of jobs in large, distributed computing clusters. The system has been in use for two years, and is sufficiently generic to be applied in two quite different domains: a Hadoop MapReduce environment and the Watson DeepQA DUCC cluster. Scalable and flexible data processing systems typically run hundreds or more of simultaneous jobs. The creation, termination, expansion and contraction of these jobs can be very dynamic and transient, and it is difficult to understand this behavior without showing its evolution over time. While traditional monitoring tools typically show either snapshots of the current load balancing or aggregate trends over time, our new visualization technique shows the behavior of each of the jobs over time in the context of the cluster, and in either a real-time or post-mortem view. Its new algorithm runs in realtime mode and can make retroactive adjustments to produce smooth layouts. Moreover, our system allows users to drill down to see details about individual jobs. The visualization has been proven useful for administrators to see the overall occupancy, trends and job allocations in the cluster, and for users to spot errors or to monitor how many resources are given to their jobs.

Abstract Distributed data stream processing applications are often characterized by data flow graphs consisting of a large number of built‐in and user‐defined operators connected via streams. These flow graphs are typically deployed on a large set of nodes. The data processing is carried out on‐the‐fly, as tuples arrive at possibly very high rates, with minimum latency. It is well known that developing and debugging distributed, multi‐threaded, and asynchronous applications, such as stream processing applications, can be challenging. Thus, without domain‐specific debugging support, developers struggle when debugging distributed applications. In this paper, we describe tools and language support to support debugging distributed stream processing applications. Our key insight is to view debugging of stream processing applications from four different, but related, perspectives. First, debugging the semantics of the application involves verifying the operator‐level composition and inspecting the flows at the logical level. Second, debugging the user‐defined operators involves traditional source‐code debugging, but strongly tied to the stream‐level interactions. Third, debugging the deployment details of the application require understanding the runtime physical layout and configuration of the application. Fourth, debugging the performance of the application requires inspecting various performance metrics (such as communication rates, CPU utilization, etc.) associated with streams, operators, and nodes in the system. In light of this characterization, we developed several tools such as adebugger‐awarecompiler and an associatedstream debugger,composition and deployment visualizers, andperformance visualizers, as well as language support, such as configuration knobs for logging and tracing, deployment configurations such as operator‐to‐process and process‐to‐node mappings, monitoring directives to inspect streams, and special sink adapters to intercept and dump streaming data to files and sockets, to name a few. We describe these tools in the context ofSpade—a language for creating distributed stream processing applications, andSystem S—a distributed stream processing middleware under development at the IBM Watson Research Center. Published in 2009 by John Wiley & Sons, Ltd.

Composite business applications typically combine characteristics of workflow and transactional applications. In order to handle multiple concurrent sessions and conversations across different nodes and over an extended time period, these applications pass dedicated identifiers inside their messages such as "OrderNumber". Debugging and understanding these conversations and the flow of conversation identifiers can be challenging in realistic business applications. We present a new heuristic algorithm to find these conversation identifiers and a method to reverse-engineer the conversations from the content of traced messages. Visualizing the results of this analysis provides a deeper understanding of the data flow, the business process and the interactions between business partners in composite business applications.

Many Java programmers believe that they do not have to worry about memory management because of automatic garbage collection. In fact, many Java programs run out of memory unexpectedly after performing a number of operations. A memory leak in Java is caused when an object that is no longer needed cannot be reclaimed because another object is still referring to it. Memory leaks can be difficult to solve, since the complexity of most programs prevents us from manually verifying the validity of every reference. In this paper we show a new methodology for finding the causes of memory leaks. We have identified a basic memory leak scenario which fits many important cases. In this scenario, we allow the programmer to identify a period of time in which temporary objects are expected to be created and released. Using this information we are able to identify objects that persist beyond this period and the references which are holding on to them. Scaling this methodology to real-world systems brings additional challenges. We propose a novel combination of visual syntax and reference pattern extraction to manage this additional complexity. We also describe how these techniques can be applied to a wider class of memory problems, including the exploration of large data structures. These techniques have been implemented and have been proven successful on large projects. Copyright © 2000 John Wiley & Sons, Ltd.

Numerous classes, complex inheritance and containment hierarchies, and diverse patterns of dynamic interaction all contribute to difficulties in understanding, reusing, debugging, and tuning large object-oriented systems. To help overcome these difficulties, we introduce novel views of the behavior of object-oriented systems and an architecture for creating and animating these views. We describe platform-independent techniques for instrumenting object-oriented programs, a language-independent protocol for monitoring their execution, and a structure for decoupling the execution of a subject program from its visualization. Case studies involving tuning and debugging of real systems are presented to demonstrate the benefits of visualization. We believe that visualization will prove to be a valuable tool for object-oriented software development.

Some multitrees with internal storage data structure are presented for the storage of graphical objects in CAD systems. Memory requirements are as low as those for the very compact quad trees without bisector lists (QWBL). Moreover, multitrees with internal storage (MTIS) are as fast as quad list quad trees (QLQT) for region search operations.>

A powerful technique (multiple storage multi-trees) for storage of geometrical data in CAD applications is presented. A significant improvement has been obtained in the field of region query performance. Memory requirements are slightly lower than for R.L. Brown's well-known multiple storage quadtrees (ibid., vol.5, no.3, p.413-9, 1986). Experimental results for the new data structure are presented. The method may prove to be very useful for VLSI design tools.>

A data structure is presented for the storage of graphical information. It is a modified multiple storage quad tree, with four lists in each leaf quad. A substantial improvement is obtained for region queries, in particular on large windows, and for tree traversal. On the other hand, only an insignificant increase of memory requirement is noticed in particular situations. The method is not complicated, so it can easily be programmed.>

Performance prediction of multiple-storage quad trees is treated. Using random graphical data models, expected values of essential parameters such as the number of internal nodes, the number of object references, and the number of lists are estimated. From these results, more practical characteristics such as memory usage and the speed of windowing operations are obtained. The accuracy of the approximations is evaluated on experimental data.>