The emergence of Superchips represents a significant advancement in next-generation AI hardware. These Superchips employ a tightly coupled heterogeneous architecture that integrates GPU and CPU on the same package, which offers unprecedented computational power. However, there has been scant research investigating how LLM training benefits from this new architecture. In this work, for the first time, we study LLM training solutions based on offloading for Superchips. We observe important differences between Superchips and traditional loosely-coupled GPU-CPU architecture, which necessitate revisiting prevailing assumptions about offloading. Based on that, we present SuperOffload, a Superchip-centric offloading system that simultaneously uses Hopper GPU, Grace CPU, and NVLink-C2C interconnect more efficiently. SuperOffload accomplishes this via a combination of techniques, such as adaptive weight offloading, bucketization repartitioning, Superchip-aware casting, speculative execution, and a highly optimized Adam optimizer for Grace CPUs. Our evaluation of SuperOffload on NVIDIA GH200 demonstrates up to 2.5× throughput improvement compared to state-of-the-art offloading-based systems, enabling training of up to 25B model on a single Superchip while achieving high training throughput. We also extend SuperOffload with ZeRO-style data parallelism and DeepSpeed-Ulysses sequence parallelism, enabling training of 13B model with sequence lengths up to 1 million tokens on 8 GH200 while achieving 55% MFU.

Computation in a typical Transformer-based large language model (LLM) can be characterized by batch size, hidden dimension, number of layers, and sequence length. Until now, system works for accelerating LLM training have focused on the first three dimensions: data parallelism for batch size, tensor parallelism for hidden size, and pipeline parallelism for model depth or layers. These widely studied forms of parallelism are not targeted or optimized for long sequence Transformer models. Given practical application needs for long sequence LLM, renewed attentions are being drawn to sequence parallelism. However, existing works in sequence parallelism are constrained by memory-communication inefficiency, limiting their scalability to long sequence large models. In this work, we introduce Ulysses, a novel, portable, and effective methodology for enabling highly efficient and scalable LLM training with extremely long sequence length. Ulysses at its core partitions input data along the sequence dimension and employs an efficient all-to-all collective communication for attention computation. Theoretical communication analysis shows that, whereas other methods incur communication overhead as sequence length increases, Ulysses maintains constant communication volume when sequence length and compute devices are increased proportionally. Furthermore, experimental evaluations show that Ulysses scales to more than 1 million context length and trains 2.5x faster with 4x longer sequence length than the existing method SOTA baseline.

Regression testing is required to check the changes in behavior whenever developers make any changes to a software system. The cost of regression testing is a major problem because developers have to frequently update dependent components to minimize security risks and potential bugs. In this paper, we report a current practice in a company that maintains an industrial simulator as a critical component of their business. The simulator automatically records all the users’ requests and the simulation results in storage. The feature provides a huge number of test cases for regression testing to developers; however, their time budget for testing is limited (i.e., at most one night). Hence, the developers need to select a small number of test cases to confirm both the simulation result and execution performance are unaffected by an update of a dependent component. In other words, the test cases should achieve high coverage while keeping diversity of execution time. To solve the problem, we have developed a clustering-based method to select test cases, using the similarity of execution traces produced by them. The developers have used the method for a half year; they recognize that the method is better than the previous rule-based method used in the company.

This work proposes RaNNC (Rapid Neural Network Connector) as middleware for automatic hybrid parallelism. In recent deep learning research, as exemplified by T5 and GPT-3, the size of neural network models continues to grow. Since such models do not fit into the memory of accelerator devices, they need to be partitioned by model parallelism techniques. Moreover, to accelerate training for huge training data, we need a combination of model and data parallelisms, i.e., hybrid parallelism. Given a model description for PyTorch without any specification for model parallelism, RaNNC automatically partitions the model into a set of subcomponents so that (1) each subcomponent fits a device memory and (2) a high training throughput for pipeline parallelism is achieved by balancing the computation times of the subcomponents. Since the search space for partitioning models can be extremely large, RaNNC partitions a model through the following three phases. First, it identifies atomic subcomponents using simple heuristic rules. Next it groups them into coarser-grained blocks while balancing their computation times. Finally, it uses a novel dynamic programming-based algorithm to efficiently search for combinations of blocks to determine the final partitions. In our experiments, we compared RaNNC with two popular frameworks, Megatron-LM (hybrid parallelism) and GPipe (originally proposed for model parallelism, but a version allowing hybrid parallelism also exists), for training models with increasingly greater numbers of parameters. In the pre-training of enlarged BERT models, RaNNC successfully trained models five times larger than those Megatron-LM could, and RaNNC's training throughputs were comparable to Megatron-LM's when pre-training the same models. RaNNC also achieved better training throughputs than GPipe on both the enlarged BERT model pre-training (GPipe with hybrid parallelism) and the enlarged ResNet models (GPipe with model parallelism) in all of the settings we tried. These results are remarkable, since RaNNC automatically partitions models without any modification to their descriptions; Megatron-LM and GPipe require users to manually rewrite the models' descriptions.

In this paper, we describe a use case applying a scientific workflow system and a distributed file system to improve the performance of telescope data processing. The application is pipeline processing of data generated by Hyper Suprime-Cam (HSC) which is a focal plane camera mounted on the Subaru telescope. In this paper, we focus on the scalability of parallel I/O and core utilization. The IBM Spectrum Scale (GPFS) used for actual operation has a limit on scalability due to the configuration using storage servers. Therefore, we introduce the Gfarm file system which uses the storage of the worker node for parallel I/O performance. To improve core utilization, we introduce the Pwrake workflow system instead of the parallel processing framework developed for the HSC pipeline. Descriptions of task dependencies are necessary to further improve core utilization by overlapping different types of tasks. We discuss the usefulness of the workflow description language with the function of scripting language for defining complex task dependency. In the experiment, the performance of the pipeline is evaluated using a quarter of the observation data per night (input files: 80 GB, output files: 1.2 TB). Measurements on strong scaling from 48 to 576 cores show that the processing with Gfarm file system is more scalable than that with GPFS. Measurement using 576 cores shows that our method improves the processing speed of the pipeline by 2.2 times compared with the method used in actual operation.

When a person gets a stroke including brain disease (e.g. cerebral bleed, cerebral infarction, subarachnoidal bleeding, etc.), the body function become worse by the hemiplegia paralysis, muscle weakness, sensory disorder, and so on. SIAS (Stroke Impairment Assessment Set) has a capability to evaluate these bodies with disabilities from multiple points of view at rehabilitation. However, the tests such as SIAS are usually evaluated by inaccurate visual observation and the results depend on the examiner. To overcome this problem, the authors have been developing a total system to get a unified decision for numerous SIAS tests by using depth sensors. In this paper, we develop the Verticality test in SIAS. This test is to evaluate the incline of trunk to lateral directions, and our system automatizes the evaluation by using Microsoft Kinect. By conducting experiments with 20 adults, we show that this system has a possibility to be used as an automatized Verticality test.

Research and development of deep learning (DL) applications often involves exhaustive trial-and-error, which demands that shared computational resources, especially GPUs, be efficiently allocated. Most DL tasks are moldable or malleable (i.e., the number of allocated GPUs can be changed before or during execution). However, conventional batch schedulers do not take advantage of DL tasks' moldability/malleability, inhibiting speedup when some GPU resources are unallocated. Another opportunity for speedup is to run multiple tasks concurrently on one GPU, which may improve the overall throughput because a single task does not always fully utilize the GPU's computational resources. We propose designing a batch scheduling system that exploits these opportunities to accelerate DL tasks. As a first step, this study conducts an extensive case study to evaluate the speedup of DL tasks when a scheduler treats them as moldable or malleable. That is, the scheduler adjusts the number of GPUs to be (or already) allocated to a task in response to the fluctuating availability of GPUs. Simulations using our real workload trace show that if the scheduler can allocate 1-4 GPUs to a task or assign 1-4 tasks to a GPU, then the average flow time of moldable/malleable DL tasks is shortened by at least 15.1 %/42.5 %, respectively, compared to a Rigid FCFS schedule in which one GPU is allocated to each task.

We propose a method for recognizing such event causalities as "smoke cigarettes" → "die of lung cancer" using background knowledge taken from web texts as well as original sentences from which candidates for the causalities were extracted. We retrieve texts related to our event causality candidates from four billion web pages by three distinct methods, including a why-question answering system, and feed them to our multi-column convolutional neural networks. This allows us to identify the useful background knowledge scattered in web texts and effectively exploit the identified knowledge to recognize event causalities. We empirically show that the combination of our neural network architecture and background knowledge significantly improves average precision, while the previous state-of-the-art method gains just a small benefit from such background knowledge.

Large-scale data analysis applications are becoming more and more prevalent in a wide variety of areas. These applications are composed of many currently available programs called analysis components. Thousands of analysis component processes are orchestrated on many compute nodes. This paper proposes a novel self-tuning framework for optimizing an application's throughput in large-scale data analysis. One challenge is developing efficient orchestration that takes into account the diversity of analysis components and the varying performances of compute nodes. In our previous work, we achieved such an orchestration to a certain degree by introducing our own middleware, which wraps each analysis component as a remote procedure call (RPC) service. The middleware also pools the processes to reduce startup overhead, which is a serious obstacle to achieving high throughput. This work tackles the remaining task of tuning the size of the analysis components' process pools to maximize the application's throughput. This is challenging because analysis components differ drastically in turnaround times and memory footprints. The size of the process pool for each type of analysis component should be set by giving consideration to these properties as well as the constraints on both the memory capacity and the processor core counts. In this work, we formulate this task as a linear programming problem and obtain the optimal pool sizes by solving it. Compared to our previous work, we significantly improved the scalability of our framework by reformulating the performance model to work on hundreds of heterogeneous nodes. We also extended the service allocation mechanism to manage the computational load on each compute node and reduce communication overhead. The experimental results show that our approach is scalable to thousands of analysis component processes running on 200 compute nodes across three clusters. Moreover, our approach significantly reduces memory footprint.

We propose a semi-supervised learning method for improving why-question answering (why-QA). The key of our method is to generate training data (question-answer pairs) from causal relations in texts such as "[Tsunamis are generated](effect) because [the ocean's water mass is displaced by an earthquake](cause)." A naive method for the generation would be to make a question-answer pair by simply converting the effect part of the causal relations into a why-question, like "Why are tsunamis generated?" from the above example, and using the source text of the causal relations as an answer. However, in our preliminary experiments, this naive method actually failed to improve the why-QA performance. The main reason was that the machine-generated questions were often incomprehensible like "Why does (it) happen?", and that the system suffered from overfitting to the results of our automatic causality recognizer. Hence, we developed a novel method that effectively filters out incomprehensible questions and retrieves from texts answers that are likely to be paraphrases of a given causal relation. Through a series of experiments, we showed that our approach significantly improved the precision of the top answer by 8% over the current state-of-the-art system for Japanese why-QA.

The importance of large-scale data analysis has shown a recent increase in a wide variety of areas, such as natural language processing, sensor data analysis, and scientific computing. Such an analysis application typically reuses existing programs as components and is often required to continuously process new data with low latency while processing large-scale data on distributed computation nodes. However, existing frameworks for combining programs into a parallel data analysis pipeline (e.g., workflow) are plagued by the following issues: (1) Most frameworks are oriented toward high-throughput batch processing, which leads to high latency. (2) A specific language is often imposed for the composition and/or such a specific structure as a simple unidirectional dataflow among constituting tasks. (3) A program used as a component often takes a long time to start up due to the heavy load at initialization, which is referred to as the startup overhead. Our solution to these problems is a remote procedure call (RPC)-based composition, which is achieved by our middleware Rapid Service Connector (RaSC). RaSC can easily wrap an ordinary program and make it accessible as an RPC service, called a RaSC service. Using such component programs as RaSC services enables us to integrate them into one program with low latency without being restricted to a specific workflow language or dataflow structure. In addition, a RaSC service masks the startup overhead of a component program by keeping the processes of the component program alive across RPC requests. We also proposed architecture that automatically manages the number of processes to maximize the throughput. The experimental results showed that our approach excels in overall throughput as well as latency, despite its RPC overhead. We also showed that our approach can adapt to runtime changes in the throughput requirements.

The on-demand provisions of cloud services create a service market, where users can dynamically select services based on such attractive criteria as price and quality. An intuitive model of a service market is a reverse auction. In the first price auction, however, a service that is cheaper and provides better quality is not necessarily selected. This causes undesirable outcomes both for users and service providers. A possible solution is the Vickrey-Clarke-Groves (VCG) mechanism, where the dominant strategy for a service provider is to report the true cost of his service. In spite of this desirable property, implementing the VCG mechanism for service composition suffers from computational cost. The calculation of payments to service providers based on the VCG mechanism requires iterative service selection, even though each service selection can be NP-hard. Approximation algorithms cannot be applied because approximate solutions do not assure the desirable property of the VCG mechanism. Thus, we model VCG payments for service markets and propose a dynamic programming (DP)-based algorithm for service selection and VCG payment calculation. Our proposed algorithm solves service selection in quasi-polynomial time and gives an exact solution. Moreover, we extend it and focus on the iterative service selection process for VCG payment calculation to improve its performance. Our series of experiments show that our proposed algorithm solves practical scale service composition.

Workflow scheduling to maximize I/O performance is one of the key issues in data-intensive, many-task computing. In our previous work, we proposed locality-aware workflow scheduling method using the Multi-Constraint Graph Partitioning. In this work, we focus on read performance of input files from the disk cache (buffer cache or page cache on main memory). In order to maximize the disk cache hit rate of input files, a LIFO-order scheduling is effective since created intermediate files may be read soon. However, LIFO policy has a disadvantage of so-called “trailing task problem.” We propose a hybrid scheduling strategy of LIFO and HRF (Highest Rank First). In our strategy, one of two policies is applied depending on the number of highest-rank tasks in the queue to avoid the problem. In addition, scheduling for the overlap of computation and I/O is proposed. We implement our scheduling strategy for the Pwrake workflow system and the Gfarm distributed file system and evaluate it by executing data-intensive workflows using a computer cluster. Our scheduling strategy improves the performance of copyfile workflow by 30% due to increase in disk cache hit rate, and the performance of Montage workflow by 12% due to increase in core utilization.

The purpose of this study is that we grasp the value of mental workload for pilot trainee (PT) using a heart rate monitor when they drill in a ship-handling simulator, and that we evaluate the effectiveness of the simulator to supply the place of a real ship based on a trend of the mental workload. The R-R interval data of subjects with a heart rate monitor have accumulated while they train for three months. The data is processed by Spline function interpolation and MEM (Maximum Entropy Method), and then we calculate LF/HF value of the data. The calculated value evaluates mental workload; we originally set a standard value and some values over the value show subject's stress condition. We summarize the number of times of subject's stress condition, and find the trend of their mental workload compared with the events of simulator scenarios for ship handling. Finally, we analyze a period when subjects feel stress, and simulator scenario where they are not good at ship handling.

The purpose of this study is evaluation of mental workload when pilot trainee practices of maneuvering simulator and evaluation of validity of it as substitution for real ship from its tendency. The subjects had trained for three months who were equipped heart rate monitor. I calculated LF/HF to process the heartbeat interval data using spline interpolation and MEM (maximum entropy method) and defined result data as mental workload. In addition, I set the ordinal reference value and evaluated reaction above this numerical value. I summarized the number of times that mental workload and grouped for mental workload tendency to compare for the contents of the corresponding scenario of simulator. In consequence, I analyzed their timing of mental workload and poor contents. As a result, reactions were differ from each them and had no uniformity for the timing or contents. Additionally in paradoxical, we understood that it is possible to get mental tendency for each to do simulator practice.

Among scheduling algorithms of scientific workflows, the graph partitioning is a technique to minimize data transfer between nodes or clusters. However, when the graph partitioning is simply applied to a complex workflow DAG, tasks in each parallel phase are not always evenly assigned to computation nodes since the graph partitioning algorithm is not aware of edge directions that represent task dependencies. Thus, we propose a new method of task assignment based on Multi-Constraint Graph Partitioning. This method relates the dimension of weight vectors to the rank of a task phase defined by traversing the task graph. Our algorithm is implemented in the Pwrake workflow system and evaluated the performance of the Montage workflow using a computer cluster. The result shows that the file size accessed from remote nodes is reduced from 88% to 14% of the total file size accessed during the workflow and that the elapsed time is reduced by 31%.

The image restoration from a blurred image is an old problem. Richardson-Lucy (RL) algorithm is a deconvolution algorithm that can restore the original image quite well if the point-spread function (PSF) is known. However, it needs to know the PSF to apply this algorithm. In this research, by restricting our problem to motion-blurred line images, we propose a blind-deconvolution algorithm by estimating PSF using Differential Evolution. As the fitness value, we define a weighted sum of two kinds of error functions. Numerical examples will show the effectiveness of the proposed algorithm.

A satisfactory solution for data-intensive web service is required to overcome the problems of web service performance and scalability. We address these problems and propose service atomization, a new approach in atomizing information analysis that tackles large scale information. It uses a measure to ease the creation and collaboration of services. We call this measure service enthalpy, which comprises enthalpy of data intensification, functionality simplification, analysis shall owing and data uniformization.

It is desirable for web applications to be reusable, for example in case a vendor wants to deliver new valuable services by using and combining existing applications quickly. One of the well-known ways of reusing web applications, Mashup, has become popular in the web development community and applied to many open web sites, as we can just combine the data retrieved from Web pages or APIs of the web applications. But in the case of the Mashup of enterprise web applications, or Enterprise Mashup, it is quite difficult since these applications are often not intended to be reusable and have no Web APIs. That means we have to create the Web APIs for the applications, which takes much work. In this paper, we describe a light-weight Web API creation methodology in order to create Web APIs for enterprise web applications much more easily than we can by extending the source code. The created Web API can connect to the web application just as a client who has an account does. So we can create a Web API for Enterprise Mashup without modifying the application at all and we can call the Web API securely using the user account information of the application. We developed the implementation of our methodology and applied it to actual applications to evaluate its effectiveness.

This paper proposes Pwrake, a parallel and distributed flexible workflow management tool based on Rake, a domain specific language for building applications in the Ruby programming language. Rake is a similar tool to make and ant. It uses a Rakefile that is equivalent to a Makefile in make, but written in Ruby. Due to a flexible and extensible language feature, Rake would be a powerful workflow management language. The Pwrake extends Rake to manage distributed and parallel workflow executions that include remote job submission and management of parallel executions. This paper discusses the design and implementation of the Pwrake, and demonstrates its power of language and extensibility of the system using a practical e-Science data-intensive workflow in astronomical data analysis on the Gfarm file system as a case study. Extending a scheduling algorithm to be aware of file locations, 20% of speed up is observed using 8 nodes (32 cores) in a PC cluster. Using two PC clusters located in different institutions, the file location aware scheduling shows scalable speedup. The extensible Pwrake is a promising workflow management tool even for wide-area data analysis.

This paper presents the results from a neural network rule extraction algorithm applied to the LED display recognition problem. We show that pruned neural networks with small number of hidden nodes and connections are able to recognize all the 10 digits from 0 to 9. Earlier work by other researchers demonstrated how symbolic fuzzy rules can be extracted from trained neural networks to solve this problem. Our rules in contrast are crisp rules, and they are obtained from smaller networks. As a result, simpler and easier to understand rules are obtained. These rules give us an insight of how neural networks differentiate one digit from the rest in LED display recognition problem.

A composite Web service designed based on abstract Web services, which define only interfaces, allows an application developer to select services required for his application only by setting endpoints for the atomic Web services. In open environment, however, the composite Web service configured in this manner may fail due to unique behaviors of the selected services. It is difficult for the designer of the composite Web service to prevent the failure because he does not know which services are selected and how they behave. On the other hand, the application developer is not authorized to modify the composite Web service due to the need to protect intellectual rights. Our solution is Service Supervision, which monitors and controls execution of composite Web services. Service Supervision makes the followings possible. 1) An application developer can control the behavior of a composite Web service by changing the execution state, even if the he is not authorized to modify the composite Web services. 2) A control pattern for coordinating Web services can be applied to various composite Web services in order to reduce the load imposed by designing control processes. In order to realize Service Supervision, we introduce meta-level control of a composite Web service. Moreover we then use the choreography to define the interaction protocols for the controls. The proposed framework is based on existing standard languages, WS-BPEL and WS-CDL. Therefore we can exploit existing tools and expertise of SOA engineers.

Multilingual expression services (MESs) are available in many multicultural fields, such as education, medical care, and disaster prevention. In Japan, for example, local governments and NPOs create parallel-texts and provide expression services in such fields. However, these services present a common difficulty: the lack of parallel-texts. Service-computing technology has the potential to allow them to collaborate. In this paper, we show how Service Oriented Architecture (SOA) can be applied to multicultural activities. As a first step, we propose an architecture, in which language resources are available as web services, and MESs can obtain parallel-texts from the language web services. To achieve this, RDF metadata are attached to parallel-texts and stored in a separate RDF repository, with which users can describe what they need in detail. This arrangement allows MESs to select only the contents they need from the shared resources and show them to users. Our implementation confirms that the proposed architecture works well with two real services in use.

Public Web services are not designed to be used with specific other Web services in a composite Web service. This leads to the following requirements for the proper control of a composite Web service; 1) adaptation to changes in Web services, 2) coordinating the contexts of internal processing, 3) flexible execution of human tasks. These controls must be applied following the policies of stakeholders, such as service providers. Some previous works proposed a method that adds processes to a composite Web service. Unfortunately, they fail to implement the controls needed and do not consider the policies of stakeholders. In this paper, we propose a meta-level architecture named Service Supervision, which controls a composite Web service based on the policies of stakeholders in order to achieve the requirements. To show the effectiveness of our architecture, we mention applications that currently apply Service Supervision to composite Web services consisting of machine translator Web services and human tasks.

This paper reports on our experiences with combining Heart of Gold and Language Grid technology to provide more language resources available on Web. Heart of Gold is known as middleware architecture for integrating deep and shallow Natural Language Processing components. The Language Grid is an infrastructure built on top of the Internet to provide distributed language services. Having Heart of Gold available as Web services in the Language Grid environment would contribute to interoperability among language services.

We consider a stochastic model for signature verification using dynamic signatures. For verification of dynamic signature data, we will use both global features and local features. Global features are such as the signature drawing time, average pen speed, the signature size, and so on. Local features are the signature shape and pen pressures evaluated locally. The stochastic model is given from the principal components of the feature vector, and two measures are defined to evaluate the distance from the center of the distribution. In the experimental study, we got a good result for authentication

In this paper, we present results of uncertain state estimation of systems that are monitored with limited accuracy. For these systems, the representation of state uncertainty as confidence intervals offers significant advantages over the more traditional approaches with probabilistic representation of noise. While the filtered-white-Gaussian noise model can be defined on grounds of mathematical convenience, its use is necessarily coupled with a hope that an estimator with good properties in idealised noise will still perform well in real noise. In this study we propose a more realistic approach of matching the noise representation to the extent of prior knowledge. Both interval and ellipsoidal representation of noise illustrate the principle of keeping the noise model simple while allowing for iterative refinement of the noise as we proceed. We evaluate one nonlinear and three linear state estimation technique both in terms of computational efficiency and the cardinality of the state uncertainty sets. The techniques are illustrated on a synthetic and a real-life system.