A Web crawler is an essential part of a search engine that procures information subsequently served by the search engine to its users. As the Web is becoming increasingly more dynamic, in addition to discovering new web pages a crawler needs to keep revisiting those already in the search engine's index, in order to keep the index fresh by picking up the pages' changed content. Determining how often to recrawl pages requires making tradeoffs based on the pages' relative importance and change rates, subject to multiple resource constraints - the limited daily budget of crawl requests on the search engine's end and politeness constraints restricting the rate at which pages can be requested from a given host. In this paper, we introduce PoliteBinaryLambdaCrawl, the first optimal algorithm for freshness crawl scheduling in the presence of politeness constraints as well as non-uniform page importance scores and the crawler's own crawl request limit. We also propose an approximation for it, stating its theoretical optimality conditions and in the process discovering a connection to an approach previously thought of as a mere heuristic for freshness crawl scheduling. We explore the relative performance of PoliteBinaryLambdaCrawl and other methods for handling politeness constraints on a dataset collected by crawling over 18.5M URLs daily over 14 weeks.

Il est naturel de s’attendre à ce que la marche aléatoire sans rebroussement mélange plus vite que la marche aléatoire simple, mais jusqu’ici, cela n’était prouvé que dans le cas des graphes réguliers. Pour analyser le cas de graphes irréguliers typiques, soit $G$ un graphe aléatoire à $n$ sommets de degrés au moins $3$ et distribués selon une loi à queue exponentielle. On détermine le temps de mélange partant du pire point de départ pour la marche aléatoire simple sur $G$, et l’on montre qu’avec grande probabilité, cette marche présente le phénomène de cutoff au temps ${\mathbf{h}}^{-1}\log n$, où ${\mathbf{h}}$ est l’entropie asymptotique de la marche aléatoire simple sur un arbre de Galton–Watson qui est une approximation locale de $G$. (Précédemment, cela n’était connu que pour des points de départ typiques.) De plus, on montre que ce temps de mélange est strictement plus grand que celui de la marche aléatoire sans rebroussement, via une comparison délicate des entropies sur l’arbre de Galton–Watson.

La dynamique de Swendsen–Wang a été proposée à la fin des années 1980 comme une alternative à la dynamique du bain-de-chaleur à un site, dans laquelle des mises à jour globales permettent à cet algorithme MCMC de passer plus vite d’un état métastable à un état de mélange idéal. Gore et Jerrum (J. Stat. Phys. 97 (1999) 67–86) ont trouvé que cette dynamique peut en fait montrer un mélange lent: ils ont montré, pour le modèle de Potts à $q\geq 3$ couleurs sur le graphe complet sur $n$ sommets au point critique $\beta_{c}(q)$, que la dynamique de Swendsen–Wang vérifie $t_{\mathrm{mix}}\geq \exp(c\sqrt{n})$. Galanis et al. (In Proc. of the 19th International Workshop on Randomization and Computation (RANDOM 2015) (2015) 815–828) a montré que $t_{\mathrm{mix}}\geq \exp(cn^{1/3})$ dans toute la fenêtre critique $(\beta_{s},\beta_{S})$ autour de $\beta_{c}$, et Blanca et Sinclair (In Proc. of the 19th International Workshop on Randomization and Computation (RANDOM 2015) (2015) 528–543) ont établit que $t_{\mathrm{mix}}\geq \exp(c\sqrt{n})$ dans la fenêtre critique pour le modèle de champs moyen FK, ce qui implique la même borne pour Swendsen–Wang grâce des estimées de comparaison connues. Dans les deux cas, une borne supérieure de $t_{\mathrm{mix}}\leq \exp(c'n)$ était connue. Dans cet article, nous montrons que le temps de mélange est vraiment exponentiel en $n$: plus précisément, $t_{\mathrm{mix}}\geq \exp (cn)$ pour la dynamique de Swendsen–Wang quand $q\geq 3$ et $\beta\in(\beta_{s},\beta_{S})$, et la même borne est vraie pour l’algorithme MCMC associé pour le modèle de champs moyen FK quand $q>2$.

Index coding has received considerable attention recently motivated in part by applications such as fast video-on-demand and efficient communication in wireless networks and in part by its connection to network coding. Optimal encoding schemes and efficient heuristics were studied in various settings, while also leading to new results for network coding such as improved gaps between linear and non-linear capacity as well as hardness of approximation. The problem of broadcasting with side information, a generalization of the index coding problem, begins with a sender and sets of users and messages. Each user possesses a subset of the messages and desires an additional message from the set. The sender wishes to broadcast a message so that on receipt of the broadcast each user can compute her desired message. The fundamental parameter of interest is the broadcast rate,$\beta $, the average communication cost for sufficiently long broadcasts. Though there have been many new nontrivial bounds on$\beta $by Bar-Yossef(2006), Lubetzky and Stav (2007), Alon(2008), and Blasiak(2011) there was no known polynomial-time algorithm for approximating$\beta $within a nontrivial factor, and the exact value of$\beta $remained unknown for all nontrivial instances. Using the information theoretic linear program introduced in Blasiak(2011), we give a polynomial-time algorithm for recognizing instances with$\beta = 2$and pinpoint$\beta $precisely for various classes of graphs (e.g., various Cayley graphs of cyclic groups). Further, extending ideas from Ramsey theory, we give a polynomial-time algorithm with a nontrivial approximation ratio for computing$\beta $. Finally, we provide insight into the quality of previous bounds by giving constructions showing separations between$\beta $and the respective bounds. In particular, we construct graphs where$\beta $is uniformly bounded while its upper bound derived from the naïve encoding scheme is polynomially worse.

The following distributed coalescence protocol was introduced by Dahlia Malkhi in 2006 motivated by applications in social networking. Initially there are n agents wishing to coalesce into one cluster via a decentralized stochastic process, where each round is as follows: Every cluster flips a fair coin to dictate whether it is to issue or accept requests in this round. Issuing a request amounts to contacting a cluster randomly chosen proportionally to its size. A cluster accepting requests is to select an incoming one uniformly (if there are such) and merge with that cluster. Empirical results by Fernandess and Malkhi suggested the protocol concludes in O(log n) rounds with high probability, whereas numerical estimates by Oded Schramm, based on an ingenious analytic approximation, suggested that the coalescence time should be super-logarithmic. Our contribution is a rigorous study of the stochastic coalescence process with two consequences. First, we confirm that the above process indeed requires super-logarithmic time w.h.p., where the inefficient rounds are due to oversized clusters that occasionally develop. Second, we remedy this by showing that a simple modification produces an essentially optimal distributed protocol: If clusters favor their smallest incoming merge request then the process does terminate in O(log n) rounds w.h.p., and simulations show that the new protocol readily outperforms the original one. Our upper bound hinges on a potential function involving the logarithm of the number of clusters and the cluster-susceptibility, carefully chosen to form a supermartingale. The analysis of the lower bound builds upon the novel approach of Schramm which may find additional applications: Rather than seeking a single parameter that controls the system behavior, instead one approximates the system by the Laplace transform of the entire cluster-size distribution.

We introduce a technique for establishing and amplifying gaps between parameters of network coding and index coding problems. The technique uses linear programs to establish separations between combinatorial and coding-theoretic parameters and applies hyper graph lexicographic products to amplify these separations. This entails combining the dual solutions of the lexicographic multiplicands and proving that this is a valid dual solution of the product. Our result is general enough to apply to a large family of linear programs. This blend of linear programs and lexicographic products gives a recipe for constructing hard instances in which the gap between combinatorial or coding-theoretic parameters is polynomially large. We find polynomial gaps in cases in which the largest previously known gaps were only small constant factors or entirely unknown. Most notably, we show a polynomial separation between linear and non-linear network coding rates. This involves exploiting a connection between matroids and index coding to establish a previously unknown separation between linear and non-linear index coding rates. We also construct index coding problems with a polynomial gap between the broadcast rate and the trivial lower bound for which no gap was previously known.

In the classical balls-and-bins paradigm, where n balls are placed independently and uniformly in n bins, typically the number of bins with at least two balls in them is ¿(n) and the maximum number of balls in a bin is ¿((log n)/(log log n)). It is well known that when each round offers k independent uniform options for bins, it is possible to typically achieve a constant maximal load if and only if k = ¿(log n). Moreover, it is possible whp to avoid any collisions between n/2 balls if k > log2n. In this work, we extend this into the setting where only m bits of memory are available. We establish a tradeoff between the number of choices k and the memory m, dictated by the quantity km/n. Roughly put, we show that for km ¿ n one can achieve a constant maximal load, while for km ¿n no substantial improvement can be gained over the case k = 1 (i.e., a random allocation). For any k = ¿(log n) and m = ¿(log2n), one can typically achieve a constant load if km = ¿(n), yet the load is unbounded if km = o(n). Similarly, if km > Cn then n/2 balls can be allocated without any collisions whp, whereas for km1-¿the optimal maximal load is ¿((log n)/(log log n)) (the same as in the case k = 1), while m = 2n suffices to ensure a constant load. Finally, we analyze non-adaptive allocation algorithms and give tight upper and lower bounds for their performance.

The following source coding problem was introduced by Birk and Kol: a sender holds a word x isin {0, 1}n, and wishes to broadcast a codeword to n receivers, Rn,..., Rn. The receiver Riis interested in xi, and has prior side information comprising some subset of the n bits. This corresponds to a directed graph G on n vertices, where i j is an edge RiRi knows the bit xj. An index code for G is an encoding scheme which enables each Ri to always reconstruct xi, given his side information. The minimal word length of an index code was studied by Bar-Yossef, Birk, Jayram, and Kol (FOCS'06). They introduced a graph parameter, minrk2(G), which completely characterizes the length of an optimal linear index code for G. They showed that in various cases linear codes attain the optimal word length, and conjectured that linear index coding is in fact always optimal. In this work, we disprove the main conjecture of Bar-Yossef, Birk, Jayram, and Kol in the following strong sense: for any epsiv > 0 and sufficiently large n, there is an n-vertex graph G so that every linear index code for G requires codewords of length at least nepsivand yet a nonlinear index code for G has a word length of ne. This is achieved by an explicit construction, which extends Alon's variant of the celebrated Ramsey construction of Frankl and Wilson. In addition, we study optimal index codes in various, less restricted, natural models, and prove several related properties of the graph parameter minrk(G).

A sender holds a word x consisting of n blocks xi, each of t bits, and wishes to broadcast a codeword to m receivers, R1,...,Rm. Each receiver Riis interested in one block, and has prior side information consisting of some subset of the other blocks. Let betatbe the minimum number of bits that has to be transmitted when each block is of length t, and let beta be the limit beta=limtrarrinfinbetat/t. Informally, beta is the average communication cost per bit in each block (for long blocks). Finding the coding rate beta, for such an informed broadcast setting, generalizes several coding theoretic parameters related to Informed Source Coding on Demand, Index Coding and Network Coding. In this work we show that usage of large data blocks may strictly improve upon the trivial encoding which treats each bit in the block independently. To this end, we provide general bounds on betat, and prove that for any constant C there is an explicit broadcast setting in which beta = 2 but beta1> C. One of these examples answers a question of . In addition, we provide examples with the following counterintuitive direct-sum phenomena. Consider a union of several mutually independent broadcast settings. The optimal code for the combined setting may yield a significant saving in communication over concatenating optimal encodings for the individual settings. This result also provides new non-linear coding schemes which improve upon the largest known gap between linear and non-linear Network Coding, thus improving the results of. The proofs are based on a relation between this problem and results in the study of Witsenhausen's rate, OR graph products, colorings of Cayley graphs, and the chromatic numbers of Kneser graphs.

The following source coding problem was introduced by Birk and Kol: a sender holds a word x epsi {0,1}n, and wishes to broadcast a codeword to n receivers, R1,..., Rnmiddot. The receiver Riis interested in x;, and has prior side information comprising some subset of the n bits. This corresponds to a directed graph G on n vertices, where ij is an edge iff Riknows the bit xj. An index code for G is an encoding scheme which enables each Rito always reconstruct Xj, given his side information. The minimal word length of an index code was studied by Bar-Yossef Birk, Jay ram and Kol. Thev introduced a graph parameter, minrk2(G), which completely characterizes the length of an optimal linear index code for G. The authors of (Z. Bar-Yossef, 2006) showed that in various cases linear codes attain the optimal word length, and conjectured that linear index coding is in fact always optimal. In this work, we disprove the main conjecture of (Z. Bar-Yossef, 2006) in the following strong sense: for any epsiv > 0 and sufficiently large n, there is an n-vertex graph G so that evety linear index code for G requires codewords of length at least n1-epsivand yet a non-linear index code for G has a word length of nepsiv. This is achieved by an explicit construction, which extends Alon's variant of the celebrated Ramsey construction of Frankl and Wilson.

The kth p‐power of a graph G is the graph on the vertex set $V(G)^k$, where two k‐tuples are adjacent iff the number of their coordinates which are adjacent in G is not congruent to 0 modulo p. The clique number of powers of G is polylogarithmic in the number of vertices; thus graphs with small independence numbers in their p‐powers do not contain large homogeneous subsets. We provide algebraic upper bounds for the asymptotic behavior of independence numbers of such powers, settling a conjecture of [N. Alon and E. Lubetzky, Combinatorica, 27 (2007), pp. 13–33] up to a factor of 2. For precise bounds on some graphs, we apply Delsarte’s linear programming bound and Hoffman’s eigenvalue bound. Finally, we show that for any nontrivial graph G, one can point out specific induced subgraphs of large p‐powers of G with neither a large clique nor a large independent set. We prove that the larger the Shannon capacity of $\overline{G}$ is, the larger these subgraphs are, and if G is the complete graph, then some p‐power of G matches the bounds of the Frankl–Wilson Ramsey construction, and is in fact a subgraph of a variant of that construction.

The independence numbers of powers of graphs have been long studied, under several definitions of graph products, and in particular, under the strong graph product. We show that the series of independence numbers in strong powers of a fixed graph can exhibit a complex structure, implying that the Shannon capacity of a graph cannot be approximated (up to a subpolynomial factor of the number of vertices) by any arbitrarily large, yet fixed, prefix of the series. This is true even if this prefix shows a significant increase of the independence number at a given power, after which it stabilizes for a while.