Training deep neural networks for large language models (LLMs) remains computationally very expensive. To mitigate this, network growing algorithms offer potential cost savings, but their underlying mechanisms are poorly understood. In this paper, we propose a theoretical framework using backward error analysis to illuminate the dynamics of mid-training network growth. Furthermore, we introduce Deep Fusion, an efficient network training approach that leverages pre-trained initializations of smaller networks, facilitating network growth from diverse sources. Our experiments validate the power of our theoretical framework in guiding the optimal use of Deep Fusion. With carefully optimized training dynamics, Deep Fusion demonstrates significant reductions in both training time and resource consumption. Importantly, these gains are achieved without sacrificing performance. We demonstrate reduced computational requirements, and improved generalization performance on a variety of NLP tasks and T5 model sizes.

We present a novel approach for detecting malicious user activity in databases. Specifically, we propose a new machine learning algorithm for detecting attacks such as a stolen user account or illegal use by a user. Our algorithm relies on two main components that examine the consistency of a user's activity and compare it with activity patterns learned from past access. The first component tests for self-consistency, to determine whether the actions performed by a user are consistent with previous patterns. This engine is based on a probabilistic model that we developed to capture a user's normal behavior. The second component checks for global-consistency, to determine whether a user's actions are consistent with the past actions of similar users. We test our algorithm on access data from SQL databases. Experimental results show that we can keep false positive rates while retaining the overall accuracy level. An outlier detection engine based on the presented methods is now included in the standard offering of IBM InfoSphere Guardium1 with positive user feedback.

We prove that for every prime $p$ there exists a (0,1)-matrix $M$ of size $t_p(n,m)\times n$, where $t_p(n,m)=O(m+\frac{m\log \frac{n}{m}}{\log \min({m,p})})$ such that every $m$ columns of $M$ are linearly independent over $\mathbb{Z}_p$, the field of integers modulo $p$ (and therefore over any field of characteristic $p$ and over the field of real numbers $\mathbb{R}$). In coding theory this matrix is a parity-check (0,1)-matrix over $\mathbb{Z}_p$ of a linear code of minimal distance m+1. Using the Hamming bound (for $p

We present our experience of using machine learning techniques over data originating from advanced meter infrastructure (AMI) systems for water consumption in a medium-size city. We focus on two new use cases that are of special importance to city authorities. One use case is the automatic identification of malfunctioning meters, with a focus on distinguishing them from legitimate non-consumption such as during periods when the household residents are on vacation. The other use case is the identification of leaks or theft in the unmetered common areas of apartment buildings. These two use cases are highly important to city authorities both because of the lost revenue they imply and because of the hassle to the residents in cases of delayed identification. Both cases are inherently complex to analyze and require advanced data mining techniques in order to achieve high levels of correct identification. Our results provide for faster and more accurate detection of malfunctioning meters as well as leaks in the common areas. This results in significant tangible value to the authorities in terms of increase in technician efficiency and a decrease in the amount of wasted, non-revenue, water.

We prove that for every prime p there exists a (0, 1)-matrix M of size tp(n, m) × n, where such that every m columns of M are linearly independent over ℤp, the field of integers modulo p (and therefore over any field of characteristic p and over the real numbers field ℝ). In coding theory this matrix is a parity-check (0, 1)-matrix over ℤp of a linear code of minimal distance m + 1. Using the Hamming bound (for p < m) and information theoretic argument (for p ≥ m) it can be shown that the above bound is tight. We show that a random tp(n, m) × n (0, 1)-matrix over ℤp satisfies the above with a high probability. This requires n·tp(n, m) random bits. To reduce the number of random bits, one can use n random variables that are m-wise independent. This gives a construction with O((m2 log n)/ log m) random bits. In this paper we use a new technique that gives for any m = nc where c is a constant, a construction that uses O(m1+∊) random bits for any constant ∊. Each row in the constructed matrix is a tensor product of a (constant) d (0, 1)-vectors of size n1/d. This solves the following open problems: Coin Weighing Problem: Suppose that n coins are given among which there are at most m counterfeit coins of arbitrary weights. There is a non-adaptive algorithm that finds the counterfeit coins and their weights in t(n, m) = O((m log n)/ log m) weighings. Previous algorithm, [CK08], solves the problem (with the same number of weighings) only for weights between n−a and nb for constants a and b and finds the counterfeit coins but not their weights. Reconstructing Graph from Additive Queries: Suppose that G is an unknown weighted graph with n vertices and m edges. There exists a non-adaptive algorithm that finds the edges of G and their weights in O(t(n, m)) additive queries. Previous algorithms, [CK08, BM09], solve the problem only for weights between n−a and nb for constants a and b and find the edges but not their weights. Signature Coding Problem: Consider n stations and at most m of them want to send messages from ℤp through an adder channel, that is, a channel that its output is the sum of the messages. Then all messages can be sent (encoded and decoded) with O(t(n, m)) transmissions. Previous algorithms, [BG07], run with the same number of transmissions only for messages in {0, 1}. Simple information theoretic arguments show that all the above bounds are tight.

In this paper, we consider the problem of reconstructing a hidden graph with m edges using additive queries.Given a graph G = (V, E) and a set of vertices S ⊆ V , an additive query, Q(S), asks for the number of edges in the subgraph induced by S. The information theoretic lower bound for the query complexity of reconstructing a graph with n vertices and m edges isIn this paper we give the first polynomial time algorithm with query complexity that matches this lower bound 1 .This solves the open problem by [S.Choi and J. Han Kim.Optimal Query Complexity Bounds for Finding Graphs.STOC, 749-758, 2008].In the paper, we actually show an algorithm for the generalized problem of reconstructing weighted graphs.In the weighted case, an additive query, Q(S), asks for the sum of weights of edges in the subgraph induces by S. The complexity of the algorithm also matches the information theoretic lower bound.

In this paper we consider the problem of reconstructing a hidden weighted hypergraph of constant rank using additive queries. We prove the following: Let $G$ be a weighted hidden hypergraph of constant rank with~$n$ vertices and $m$ hyperedges. For any $m$ there exists a non-adaptive algorithm that finds the edges of the graph and their weights using $$ O\left(\frac{m\log n}{\log m}\right) $$ additive queries. This solves the open problem in [S. Choi, J. H. Kim. Optimal Query Complexity Bounds for Finding Graphs. {\em STOC}, 749--758, 2008]. When the weights of the hypergraph are integers that are less than $O(poly(n^d/m))$ where $d$ is the rank of the hypergraph (and therefore for unweighted hypergraphs) there exists a non-adaptive algorithm that finds the edges of the graph and their weights using $$ O\left(\frac{m\log \frac{n^d}{m}}{\log m}\right). $$ additive queries. Using the information theoretic bound the above query complexities are tight.