The Nyström method is a popular low-rank approximation technique for large matrices that arise in kernel methods and convex optimization. Yet, when the data exhibits heavy-tailed spectral decay, the effective dimension of the problem often becomes so large that even the Nyström method may be outside of our computational budget. To address this, we propose Block-Nyström, an algorithm that injects a block-diagonal structure into the Nyström method, thereby significantly reducing its computational cost while recovering strong approximation guarantees. We show that Block-Nyström can be used to construct improved preconditioners for second-order optimization, as well as to efficiently solve kernel ridge regression for statistical learning over Hilbert spaces. Our key technical insight is that, within the same computational budget, combining several smaller Nyström approximations leads to stronger tail estimates of the input spectrum than using one larger approximation. Along the way, we provide a novel recursive preconditioning scheme for efficiently inverting the Block-Nyström matrix, and provide new statistical learning bounds for a broad class of approximate kernel ridge regression solvers.

Gaussian processes (GPs) play an essential role in biostatistics, scientific machine learning, and Bayesian optimization for their ability to provide probabilistic predictions and model uncertainty. However, GP inference struggles to scale to large datasets (which are common in modern applications), since it requires the solution of a linear system whose size scales quadratically with the number of samples in the dataset. We propose an approximate, distributed, accelerated sketch-and-project algorithm ($\texttt{ADASAP}$) for solving these linear systems, which improves scalability. We use the theory of determinantal point processes to show that the posterior mean induced by sketch-and-project rapidly converges to the true posterior mean. In particular, this yields the first efficient, condition number-free algorithm for estimating the posterior mean along the top spectral basis functions, showing that our approach is principled for GP inference. $\texttt{ADASAP}$ outperforms state-of-the-art solvers based on conjugate gradient and coordinate descent across several benchmark datasets and a large-scale Bayesian optimization task. Moreover, $\texttt{ADASAP}$ scales to a dataset with $> 3 \cdot 10^8$ samples, a feat which has not been accomplished in the literature.

We show that, for finite-sum minimization problems, incorporating partial second-order information of the objective function can dramatically improve the robustness to mini-batch size of variance-reduced stochastic gradient methods, making them more scalable while retaining their benefits over traditional Newton-type approaches. We demonstrate this phenomenon on a prototypical stochastic second-order algorithm, called Mini-Batch Stochastic Variance-Reduced Newton ($\texttt{Mb-SVRN}$), which combines variance-reduced gradient estimates with access to an approximate Hessian oracle. In particular, we show that when the data size $n$ is sufficiently large, i.e., $n\gg \alpha^2\kappa$, where $\kappa$ is the condition number and $\alpha$ is the Hessian approximation factor, then $\texttt{Mb-SVRN}$ achieves a fast linear convergence rate that is independent of the gradient mini-batch size $b$, as long $b$ is in the range between $1$ and $b_{\max}=O(n/(\alpha \log n))$. Only after increasing the mini-batch size past this critical point $b_{\max}$, the method begins to transition into a standard Newton-type algorithm which is much more sensitive to the Hessian approximation quality. We verify this empirically on benchmark tasks showing that, after tuning the step size, the convergence rate of $\texttt{Mb-SVRN}$ remains fast for a wide range of mini-batch sizes, and the dependence of the phase transition point $b_{\max}$ on the Hessian approximation factor $\alpha$ agrees with our theory.

Matrix sketching is a powerful tool for reducing the size of large data matrices. Yet there are fundamental limitations to this size reduction when we want to recover an accurate estimator for a task such as least square regression. We show that these limitations can be circumvented in the distributed setting by designing sketching methods that minimize the bias of the estimator, rather than its error. In particular, we give a sparse sketching method running in optimal space and current matrix multiplication time, which recovers a nearly-unbiased least squares estimator using two passes over the data. This leads to new communication-efficient distributed averaging algorithms for least squares and related tasks, which directly improve on several prior approaches. Our key novelty is a new bias analysis for sketched least squares, giving a sharp characterization of its dependence on the sketch sparsity. The techniques include new higher moment restricted Bai-Silverstein inequalities, which are of independent interest to the non-asymptotic analysis of deterministic equivalents for random matrices that arise from sketching.

In online advertising, response prediction is the problem of estimating the probability that an advertisement is clicked when displayed on a content publisher's webpage. In this paper, we show how response prediction can be viewed as a problem of matrix completion, and propose to solve it using matrix factorization techniques from collaborative filtering (CF). We point out the two crucial differences between standard CF problems and response prediction, namely the requirement of predicting probabilities rather than scores, and the issue of confidence in matrix entries. We address these issues using a matrix factorization analogue of logistic regression, and by applying a principled confidence-weighting scheme to its objective. We show how this factorization can be seamlessly combined with explicit features or side-information for pages and ads, which let us combine the benefits of both approaches. Finally, we combat the extreme sparsity of response prediction data by incorporating hierarchical information about the pages and ads into our factorization model. Experiments on three very large real-world datasets show that our model outperforms current state-of-the-art methods for response prediction.

In Contextual advertising, textual ads relevant to the content in a webpage are embedded in the page. Content keywords are extracted offline by crawling webpages and then stored in an index for fast serving. Given a page, ad selection involves index lookup, computing similarity between the keywords of the page and those of candidate ads and returning the top-k scoring ads. In this approach, ad relevance can suffer in two scenarios. First, since page-ad similarity is computed using keywords extracted only from that particular page, a few non pertinent keywords can skew ad selection. Second, requesting page may not be present in the index but we still need to serve relevant ads.

In Contextual advertising, textual ads relevant to the content in a webpage are embedded in the page. Content keywords are extracted offline by crawling webpages and then stored in an index for fast serving. Given a page, ad selection involves index lookup, computing similarity between the keywords of the page and those of candidate ads and returning the top-k scoring ads. In this approach, there is a tradeoff between relevance and index size where better relevance can be achieved if there are no limits on the index size. However, the assumption of unlimited index size is not practical due to the large number of pages on the Web and stringent requirements on the serving latency. Secondly, page visits on the web follows power-law distribution where a significant proportion of the pages are visited infrequently, also called the tail pages. Indexing tail pages is not efficient given that these pages are accessed very infrequently.

Presence of duplicate documents in the World Wide Web adversely affects crawling, indexing and relevance, which are the core building blocks of web search. In this paper, we present a set of techniques to mine rules from URLs and utilize these rules for de-duplication using just URL strings without fetching the content explicitly. Our technique is composed of mining the crawl logs and utilizing clusters of similar pages to extract transformation rules, which are used to normalize URLs belonging to each cluster. Preserving each mined rule for de-duplication is not efficient due to the large number of such rules. We present a machine learning technique to generalize the set of rules, which reduces the resource footprint to be usable at web-scale. The rule extraction techniques are robust against web-site specific URL conventions. We compare the precision and scalability of our approach with recent efforts in using URLs for de-duplication. Experimental results demonstrate that our approach achieves 2 times more reduction in duplicates with only half the rules compared to the most recent previous approach. Scalability of the framework is demonstrated by performing a large scale evaluation on a set of 3 Billion URLs, implemented using the MapReduce framework.

Presence of duplicate documents in the World Wide Web adversely affects crawling, indexing and relevance, which are the core building blocks of web search. In this paper, we present a set of techniques to mine rules from URLs and utilize these learnt rules for de-duplication using just URL strings without fetching the content explicitly. Our technique is composed of mining the crawl logs and utilizing clusters of similar pages to extract specific rules from URLs belonging to each cluster. Preserving each mined rules for de-duplication is not efficient due to the large number of specific rules. We present a machine learning technique to generalize the set of rules, which reduces the resource footprint to be usable at web-scale. The rule extraction techniques are robust against web-site specific URL conventions. We demonstrate the effectiveness of our techniques through experimental evaluation.

Abstract Wireless access points (APs) act as bridges between wired and wireless networks. Since the actually available bandwidth in wireless networks is much smaller than the bandwidth in wired networks, there is a disparity in channel capacity which makes the access point a significant network congestion point in the downstream direction. A current architectural trend in wireless local area networks (WLAN) is to move functionality from APs to a centralized gateway in order to reduce cost and improve features. In this paper, we study the use of RED, a well known active queue management (AQM) scheme, and explicit congestion notification (ECN) to handle bandwidth disparity between the wired and the wireless interface of an access point. Then, we propose the Proxy‐RED scheme, as a solution for reducing the AQM overhead from the access point. Simulations‐based performance analysis indicates that the proposed Proxy‐RED scheme improves the overall performance of a network. In particular, the Proxy‐RED scheme significantly reduces packet loss rate and improves goodput for a small buffer, and minimizes delay for a large buffer size. Copyright © 2006 John Wiley & Sons, Ltd.

We propose a comprehensive solution to handle memory-overflow problems in multitasking embedded systems thereby improving their reliability and availability. In particular, we propose two complementary techniques to address two significant causes of memory-overflow problems. The first cause is errors in estimating appropriate stack and heap memory requirement. Our first technique, called shared supplementary memory (SSM), exploits the fact that the probability of multiple tasks requiring more than their estimated amount of memory concurrently is low. Using analytical model and simulations, we show that reliability can be considerably improved when SSM is employed. Furthermore, for the same reliability, SSM reduces total memory requirement by as much as 29.31% The second cause is the presence of coding Mandelbugs, which can cause abnormal memory requirement. To address this, we propose a novel technique, called opportunistic micro-rejuvenation, which when combined with SSM, provide several advantages: preventing critical-time outage, resource frugality and dependability enhancement.

Secure Real-time Transport Protocol (SRTP) provides confidentiality, authentication, integrity and replay protection for secure media transport in VoIP. However, the overhead of HMAC-SHA1 incurred per packet makes SRTP susceptible to flooding based Denial-of-Service attack. In this paper, we present a class of schemes to increase the DoS tolerance in SRTP. The central idea is to add a light-weight authentication mechanism on top of SRTP. This mechanism is used to efficiently discard illegitimate packets early on in the face of a DoS attack. Analysis shows that substantially larger traffic flood can be handled with the proposed enhancements.

Voice-over-IP (VoIP) systems are gaining in popularity as the technology for transmitting voice traffic over IP networks. As the popularity of VoIP systems increases, they are being subjected to different kinds of intrusions some of which are specific to such systems and some of which follow a general pattern. VoIP systems pose several new challenges to intrusion detection system (IDS) designers. First, these systems employ multiple protocols for call management (e.g., SIP) and data delivery (e.g., RTP). Second, the systems are distributed in nature and employ distributed clients, servers and proxies. Third, the attacks to such systems span a large class, from denial of service to billing fraud attacks. Finally, the systems are heterogeneous and typically under several different administrative domains. In this paper, we propose the design of an intrusion detection system targeted to VoIP systems, called SCIDIVE (pronounced "Skydive"). SCIDIVE is structured to detect different classes of intrusions, including, masquerading, denial of service, and media stream-based attacks. It can operate with both classes of protocols that compose VoIP systems - call management protocols (CMP), e.g., SIP, and media delivery protocols (MDP), e.g., RTP. SCIDIVE proposes two abstractions for VoIP IDS - stateful detection and cross-protocol detection. Stateful detection denotes assembling state from multiple packets and using the aggregated state in the rule-matching engine. Cross protocol detection denotes matching rules that span multiple protocols. SCIDIVE is demonstrated on a sample VoIP system that comprises SIP clients and SIP proxy servers with RTP as the data delivery protocol. Four attack scenarios are created and the accuracy and the efficiency of the system evaluated with rules meant to catch these attacks.

Wireless access points act as bridges between wired and wireless networks. Since the actually available bandwidth in wireless networks is much smaller than the bandwidth in wired networks, there is a disparity in channel capacity which makes the access point a significant network congestion point in the downstream direction. A current architectural trend in wireless local area networks (WLAN) is to move functionality from access points to a centralized gateway in order to reduce cost and improve features. We study the use of RED, a well known active queue management (AQM) scheme, and explicit congestion notification (ECN) to handle bandwidth disparity between the wired and the wireless interface of an access point Then, we propose the proxy-RED scheme, as a solution for reducing the AQM overhead from the access point. Simulations-based performance analysis indicates that the proposed proxy-RED scheme improves overall performance of the network. In particular, the proxy-RED scheme significantly reduces packet loss rate and improves goodput for a small buffer, and minimizes delay for a large buffer size.

The presence of physical obstacles and radio interfer-ence results in the so called “shadow regions ” in wireless networks. When a mobile station roams into a shadow re-gion, it loses its network connectivity. In cellular networks, in order to minimize the connection unreliability, careful cell planning is required to prevent the occurrance of the shadow regions in the first place. In 802.11b/g wireless LANs, however, due to the limited frequency spectrum, it is not always possible to prevent a shadow region by adding another cell at a different frequency. Our contribution in this paper is to propose the alternate approach of tolerating the existence of “shadow regions ” as opposed to prevention in order to enhance the connection dependability. A redundant access point (AP) is placed in

In this paper, we propose a metric for measuring the utilization of an IEEE 802.11b wireless network and outline how this metric can be accurately estimated using data that is readily available in most access points. We furthermore describe how this metric can be used to perform admission control for VoIP traffic and describe experiences with a prototype implementation. Admission control for VoIP traffic in 802.11 networks is necessary since the number of simultaneous VoIP connections in a single cell of an 802.11 network is very small.

In this paper, we study the inherent limitations of the 802.11 (a/b) distributed coordination function (DCF) in supporting VoIP calls over a wireless LAN. Specifically, we evaluate the upper bound on the number of simultaneous VoIP calls that can be placed in a single cell of an 802.11 (a/b) network. Making one additional VoIP call in that cell would degrade the quality of all VoIP call. The upper bound is calculated as a function of the choice of VoIP codec and the length of the audio payload. As an example, when a G711 codec with 20 millisecond audio payload is used, an 802.11b cell can support only 3 to 12 simultaneous VoIP calls. The actual number depends on the effective transmission rate of the wireless station, which for 802.11b can be 1 Mbps, 5.5 Mbps and 11 Mbps. We also study the effect of spatial distribution of the wireless stations on the upper bound which is the dominant factor in determining the effective transmission rate of a station.

We present experimental studies on the throughput of IEEE 802.11b wireless networks for UDP and VoIP traffic. Our experiments show that the maximum data throughput of a single station sending out UDP traffic is 6.1 Mbps. The maximum number of VoIP calls in a single cell of an IEEE 802.11b network is six if the ITU G711a-Law codec is used with 10 milliseconds of audio per RTP packet. The experiments also show that the effective available bandwidth in the wireless network is reduced by ongoing VoIP connections. Specifically, for the above codec settings, each VoIP connection reduces the bandwidth available for data traffic by 900 kbps.

In this paper we describe the "wireless access server (WAS)". WAS is targeted towards providing QoS and access control features for wireless LAN, specifically 802.11 networks. It is well known that these aspects have significant deficiencies in the widely deployed 802.11 networks. Our work is complementary to the proposed drafts by IEEE's 802.11e and 802.11i Working Groups, which also aim to alleviate some of the limitations in QoS and security. We describe the architecture and components of the server and outline the QoS and access control functionality it provides.

Network survivability reflects the ability of a network to continue to function during and after failures. Our purpose in this paper is to propose a quantitative approach to evaluate network survivability. We perceive the network survivability as a composite measure consisting of both network failure duration and failure impact on the network. A wireless ad-hoc network is analyzed as an example, and the excess packet loss due to failures (ELF) is taken as the survivability performance measure. To obtain ELF, we adopt a two phase approach consisting of the steady-state availability analysis and transient performance analysis. Assuming Markovian property for the system, this measure is obtained by solving a set of Markov models. By utilizing other analysis paradigms, our approach in this paper may also be applied to study the survivability performance of more complex systems.

The phenomenon of software aging refers to the accumulation of errors during the execution of the software which eventually results in it's crash/hang failure. A gradual performance degradation may also accompany software aging. Pro-active fault management techniques such as "software rejuvenation" (Y. Huang et al., 1995) may be used to counteract aging if it exists. We propose a methodology for detection and estimation of aging in the UNIX operating system. First, we present the design and implementation of an SNMP based, distributed monitoring tool used to collect operating system resource usage and system activity data at regular intervals, from networked UNIX workstations. Statistical trend detection techniques are applied to this data to detect/validate the existence of aging. For quantifying the effect of aging in operating system resources, we propose a metric: "estimated time to exhaustion", which is calculated using well known slope estimation techniques. Although the distributed data collection tool is specific to UNIX, the statistical techniques can be used for detection and estimation of aging in other software as well.

Checkpointing and roll-back recovery is a well known technique for recovering from software process failures. Analytical models have been developed for computing the completion time of processes that use various checkpointing strategies such as periodic checkpointing, random checkpointing etc. In this paper, we show that with active replication of processes, a strategy that uses a mechanism we call checkpoints-on-demand will result in an expected completion time smaller than that can be achieved with traditional schemes that use periodic checkpoints. With checkpoints-on-demand, when a process fails, it is recovered from an induced checkpoint taken of a replica of the process. Recovery of persistent server processes through state-transfer from a replica has been proposed in the context of group communication systems and in the process cloning approach of the Delta-4 architecture. But it has not been previously proposed and analyzed as a mechanism for reducing the expected completion time of a long running process.

Preventive maintenance of operational software systems, a novel technique for software fault tolerance, is used specifically to counteract the phenomenon of software "aging". However, it incurs some overhead. The necessity to do preventive maintenance, not only in general purpose software systems of mass use, but also in safety-critical and highly available systems, clearly indicates the need to follow an analysis based approach to determine the optimal times to perform preventive maintenance. In this paper, we present an analytical model of a software system which serves transactions. Due to aging, not only the service rate of the software decreases with time, but also the software itself experiences crash/hang failures which result in its unavailability. Two policies for preventive maintenance are modeled and expressions for resulting steady state availability, probability that an arriving transaction is lost and an upper bound on the expected response time of a transition are derived. Numerical examples are presented to illustrate the applicability of the models.

With the increased attention ATM is receiving to meet the needs of a wide variety of applications, tools are needed to help a network designer focus on the design at hand, rather than to spend time exhaustively learning the tools themselves. This is the concept behind IDEA. This paper introduces "modeling engines" chosen to be integrated into IDEA and presents a user interface for networking design. The first objective for IDEA is to demonstrate its usefulness for dependability modeling. Later objectives include incorporating performance and performability modeling.

Checkpointing with rollback-recovery is a well known technique to reduce the completion time of a program in the presence of failures. While checkpointing is corrective in nature, rejuvenation refers to preventive maintenance of software aimed to reduce unexpected failures mostly resulting from the "aging" phenomenon. In this paper, we show how both these techniques may be used together to further reduce the expected completion time of a program. The idea of using checkpoints to reduce the amount of rollback upon a failure is taken a step further by combining it with rejuvenation. We derive the equations for expected completion time of a program with finite failure free running time for the following three cases when; (a) neither checkpointing nor rejuvenation is employed, (b) only checkpointing is employed, and finally (c) both checkpointing and rejuvenation are employed.We also present numerical results for Weibull failure time distribution for the above three cases and discuss optimal checkpointing and rejuvenation that minimizes the expected completion time. Using the numerical results, some interesting conclusions are drawn about benefits of these techniques in relation to the nature of failure distribution.

In a client-server type system, the server software is required to run continuously for very long periods. Due to repeated and potentially faulty usage by many clients, such software "ages" with time and eventually fails. (Huang et al., 1995) proposed a technique called "software rejuvenation" in which the software is periodically stopped and then restarted in a "robust" state after proper maintenance. This "renewal" of software prevents (or at least postpones) the crash failure. As the time lost (or the cost incurred) due to the software failure is typically more than the time lost (or the cost incurred) due to rejuvenation, the technique reduces the expected unavailability of the software. We present a quantitative analysis of software rejuvenation. The behavior of the system is represented through a Markov Regenerative Stochastic Petri Net (MRSPN) model which is solved both for steady state as well as transient conditions. We provide a closed-form analytical solution for the steady state expected down time (and the expected cost incurred) due to system unavailability. We also evaluate the optimal rejuvenation interval which minimizes the expected unavailability of the software.