We analyze the performance of a genetic algorithm (GA) we call Culling, and a variety of other algorithms, on a problem we refer to as the Additive Search Problem (ASP). We show that the problem of learning the Ising perceptron is reducible to a noisy version of ASP. Noisy ASP is the first problem we are aware of where a genetic-type algorithm bests all known competitors. We generalize ASP to k-ASP to study whether GAs will achieve "implicit parallelism" in a problem with many more schemata. GAs fail to achieve this implicit parallelism, but we describe an algorithm we call Explicitly Parallel Search that succeeds. We also compute the optimal culling point for selective breeding, which turns out to be independent of the fitness function or the population distribution. We also analyze a mean field theoretic algorithm performing similarly to Culling on many problems. These results provide insight into when and how GAs can beat competing methods.

The use of dynamically-dispatched procedure calls is a key mechanism for writing extensible and flexible code in object-oriented languages. Unfortunately, dynamic dispatching imposes a runtime performance penalty. Some recent implementations of pure object-oriented languages have utilized profile-guided receiver class prediction to reduce this performance penalty, and some researchers have argued for applying receiver class prediction in hybrid languages like C++. We performed a detailed examination of the dynamic profiles of eight large object-oriented applications written in C++ and Cecil, determining that the receiver class distributions are strongly peaked and stable across both inputs and program versions through time. We describe techniques for gathering and manipulating profile information at varying degrees of precision, particularly in the presence of optimizations such as inlining. Our implementation of profile-guided receiver class prediction improves the performance of large Cecil applications by more than a factor of two over solely static optimizations.

One way to speed up convergence in a large optimization problem is to introduce a smaller, approximate version of the problem at a coarser scale and to alternate between relaxation steps for the fine-scale and coarse-scale problems. Such an optimization method for neural networks governed by quite general objective functions is presented. At the coarse scale, there is a smaller approximating neural net which, like the original net, is nonlinear and has a nonquadratic objective function. The transitions and information flow from fine to coarse scale and back do not disrupt the optimization, and the user need only specify a partition of the original fine-scale variables. Thus, the method can be applied easily to many problems and networks. There is generally about a fivefold improvement in estimated cost under the multiscale method. In the networks to which it was applied, a nontrivial speedup by a constant factor of between two and five was observed, independent of problem size. Further improvements in computational cost are very likely to be available, especially for problem-specific multiscale neural net methods.

A multiscale optimization method for neural networks governed by quite general objective functions is presented. At the coarse scale, there is a smaller, approximating neural net. Like the original net, it is nonlinear and has a nonquadratic objective function, so the coarse-scale net is a more accurate approximation than a quadratic objective would be. The transitions and information flow form fine to coarse scale and back do not disrupt the optimization. The problem need not involve any geometric domain; all that is required is a partition of the original fine-scale variables. Given this partition, the rest of the multiscale optimization method requires no problem-specific design effort on the part of the user, since the mapping between coarse and fine scales is determined. Thus, the method can be applied easily to many problems and networks. Positive experimental results including cost comparisons are shown