Quantitative Finance (QF) utilizes increasingly sophisticated mathematic models and advanced computer techniques to predict the movement of global markets, and price the derivatives and other assets. Being able to react quickly and intelligently to fast-changing markets is a decisive success factor for trading companies. To date, the rise of QF requires an integrated toolchain of enabling technologies to carry out complex event processing on the explosive growth and diversified forms of market metadata, in pursuit of a microsecond latency on an Exabyte-level dataset. Inspired by this, we present a data-driven execution paradigm that untangles the dependencies of complex processing events and integrate the paradigm with a big data infrastructure that streams time series data. This integrated platform is termed as the QuantCloud platform. Essentially, QuantCloud executes the complex event processing in a data-driven mode and manages large amounts of diversified market data in a data-parallel mode. To show its practicability and performance, we develop a prototype and benchmark by applying real-world QF research models on the New York Stock Exchange (NYSE) data. Using this prototype, we demonstrate this platform with an application to: (i) data cleaning and aggregating (including the computing of logarithmic returns from tick data and the finding the medians of grouped data) and (ii) data modeling: the autoregressive-moving average (ARMA) model. The performance results show that (a) this platform obtains a high throughput (usually in the order of millions of tick messages per second) and a sub-microsecond latency; (b) it fully executes data-dependent tasks through a data-driven execution; and (c) it implements a modular design approach for rapidly developing these data-crunching methods and QF research models. This platform resulting from an aggregated effort of the data-driven execution and big data infrastructure, offers the financial engineers with new insights and enhanced capabilities for effective and efficient incorporation of big data complex event processing technologies in their workflow.

Cloud services have been widely employed in IT industry and scientific research. By using Cloud services users can move computing tasks and data away from local computers to remote datacenters. By accessing Internet-based services over lightweight and mobile devices, users deploy diversified Cloud applications on powerful machines. The key drivers towards this paradigm for the scientific computing field include the substantial computing capacity, on-demand provisioning and cross-platform interoperability. To fully harness the Cloud services for scientific computing, however, we need to design an application-specific platform to help the users efficiently migrate their applications. In this, we propose a Cloud service platform for symbolic-numeric computation-SNC. SNC allows the Cloud users to describe tasks as symbolic expressions through C/C++, Python, Java APIs and SNC script. Just-In-Time (JIT) compilation through using LLVM/JVM is used to compile the user code to the machine code. We implemented the SNC design and tested a wide range of symbolic-numeric computation applications (including nonlinear minimization, Monte Carlo integration, finite element assembly and multibody dynamics) on several popular cloud platforms (including the Google Compute Engine, Amazon EC2, Microsoft Azure, Rackspace, HP Helion and VMWare vCloud). These results demonstrate that our approach can work across multiple cloud platforms, support different languages and significantly improve the performance of symbolic-numeric computation using cloud platforms. This offered a way to stimulate the need for using the cloud computing for the symbolic-numeric computation in the field of scientific research.

In this paper, we present the QuantCloud infrastructure, designed for performing big data analytics in modern quantitative finance. Through analyzing market observations, quantitative finance (QF) utilizes mathematical models to search for subtle patterns and inefficiencies in financial markets to improve prospective profits. To discover profitable signals in anticipation of volatile trading patterns amid a global market, analytics are carried out on Exabyte-scale market metadata with a complex process in pursuit of a microsecond or even a nanosecond of data processing advantage. This objective motivates the development of innovative tools to address challenges for handling high volume, velocity, and variety investment instruments. Inspired by this need, we developed QuantCloud by employing large-scale SSD-backed datastore, various parallel processing algorithms, and portability in Cloud computing. QuantCloud bridges the gap between model computing techniques and financial data-driven research. The large volume of market data is structured in an SSD-backed datastore, and a daemon reacts to provide the Data-on-Demand services. Multiple client services process user requests in a parallel mode and query on-demand datasets from the datastore through Internet connections. We benchmark QuantCloud performance on a 40-core, 1TB-memory computer and a 5-TB SSD-backed datastore. We use NYSE TAQ data from the fourth quarter of 2014 as our market data. The results indicate data-access application latency as low as 3.6 nanoseconds per message, sustained throughput for parallel data processing as high as 74 million messages per second, and completion of 11 petabyte-level data analytics within 53 minutes. Our results demonstrate that the aggregated contributions of our infrastructure, parallel algorithms, and sophisticated implementations offer the algorithmic trading and financial engineering community new hope and numeric insights for their research and development.

We present a network configuration scheme exampled by a hardwired 64-node parallel computer with interconnection topologies of 2D Torus(8 x 8), 2D iBT(8 x 8; b = 〈2〉) and 3D Torus(4 x 4 x 4) reconfigurated by software. On this platform, we evaluate the performance ratios through adding bypass links (iBT) or torus links (3D Torus) to the base 2D Torus. We benchmark the three network topologies on elementary and collective communications as well as parallel applications including NAS parallel benchmark, NAMD, high performance Linpack and HPC challenge benchmark suites. Through comparative analysis of the relative performances, we reveal and, in several cases, reaffirm that strategically adding links could greatly improve the performance in 94 percent tested application cases; the iBT network outperforms others in more than half of the tested cases; and our network configuration scheme would be an alternative, better than the classical network scheme, for constructing a parallel processing system with a variety of application patterns.

As an application of interlaced bypass torus (iBT) interconnection networks, a 6D mesh/iBT network has been formed by replacing the 3D torus network in a 6D mesh/torus interconnect (Tofu) with a 3D iBT network, helping further reduce latencies. However, the routing algorithms good for the torus-based systems such as Blue Gene series may deadlock for the new network. This work proposes three deadlock-free routing algorithms for iBT networks and an optimizing method. An iBT network with these routings is simulated and compared with a 3D torus and a 4D torus network. Results for all-to-all communications, when applying the optimal routing algorithm, show a link utilization of 96% of the theoretical peak. An iBT prototyped system is also built and preliminarily tested.

A new 6D mesh/torus (Tofu) interconnect is implemented in the K Computer for achieving a performance of more than 10 petaflops. As expected, among its many interesting properties, such interconnect has large variations of latencies for communications within a node-group (intra-node-group) or between them (inter-node-group). Inspired by their works while recognizing the superior features of the recently published interlaced bypass torus (iBT) network that can help reduce such variations, we propose a new 6D mesh/iBT interconnect by replacing the 3D torus of Tofu by a 3D iBT. Our new interconnect suppresses the latencies for inter-node-group communications to dramatically improve the overall performance for the entire network. Adapting an industrial standard network simulator NS-3, we study different communication patterns of the added iBT networks under different bypass schemes. We find that the simulated average latencies of our network are 40% less than those of the Tofu interconnect for all tested patterns.

Broadcast algorithms for the interlaced bypass torus networks (iBT networks) are introduced to balance the all-port bandwidth efficiency and to avoid congestion in multidimensional cases. With these algorithms, we numerically analyze the dependencies of the broadcast efficiencies on various packet-sending patterns, bypass schemes, network sizes, and dimensionalities and then strategically tune up the configurations for minimizing the broadcast steps. Leveraging on such analysis, we compare the performance of networks with one million nodes between two cases: one with an added fixed-length bypass links and the other with an added torus dimension. A case study of iBT(10002;b = (8,32)) and Torus(1003) shows that the former improves the diameter, average node-to-node distance, rectangular and global broadcasts over the latter by approximately 80 percent. It is reaffirmed that strategically interlacing short bypass links and methodically utilizing these links is superior to adding dimensionalities to torus in achieving shorter diameter, average node-to-node distances and faster broadcasts.

We introduce a new technique for generating more efficient networks by systematically interlacing bypass rings to torus networks (iBT networks). The resulting network can improve the original torus network by reducing the network diameter, node-to-node distances, and by increasing the bisection width without increasing wiring and other engineering complexity. We present and analyze the statement that a 3D iBT network proposed by our technique outperforms 4D torus networks of the same node degree. We found that interlacing rings of sizes 6 and 12 to all three dimensions of a torus network with meshes 30 × 30 × 36 generate the best network of all possible networks, including 4D torus and hypercube of approximately 32,000 nodes. This demonstrates that strategically interlacing bypass rings into a 3D torus network enhances the torus network more effectively than adding a fourth dimension, although we may generalize the claim. We also present a node-to-node distance formula for the iBT networks.