Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are insufficient for instilling self-correction behavior. In particular, we observe that training via SFT either suffers from a distribution mismatch between the training data and the model's own responses or implicitly prefers only a certain mode of correction behavior that is often not effective at test time. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction strategy that is effective at test time as opposed to simply fitting high-reward responses for a given prompt. This regularization prescribes running a first phase of RL on a base model to generate a policy initialization that is less susceptible to collapse and then using a reward bonus to amplify self-correction during training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on the MATH and HumanEval benchmarks.

Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a vast space of held-out environment dynamics, our adaptive agent (AdA) displays on-the-fly hypothesis-driven exploration, efficient exploitation of acquired knowledge, and can successfully be prompted with first-person demonstrations. Adaptation emerges from three ingredients: (1) meta-reinforcement learning across a vast, smooth and diverse task distribution, (2) a policy parameterised as a large-scale attention-based memory architecture, and (3) an effective automated curriculum that prioritises tasks at the frontier of an agent’s capabilities. We demonstrate characteristic scaling laws with respect to network size, memory length, and richness of the training task distribution. We believe our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.

Traditional automated theorem proving systems for first-order logic depend on speed-optimized search and many handcrafted heuristics designed to work over a wide range of domains. Machine learning approaches in the literature either depend on these traditional provers to bootstrap themselves, by leveraging these heuristics, or can struggle due to limited existing proof data. The latter issue can be explained by the lack of a smooth difficulty gradient in theorem proving datasets; large gaps in difficulty between different theorems can make training harder or even impossible. In this paper, we adapt the idea of hindsight experience replay from reinforcement learning to the automated theorem proving domain, so as to use the intermediate data generated during unsuccessful proof attempts. We build a first-order logic prover by disabling all the smart clause-scoring heuristics of the state-of-the-art E prover and replacing them with a clause-scoring neural network learned by using hindsight experience replay in an incremental learning setting. Clauses are represented as graphs and presented to transformer networks with spectral features. We show that provers trained in this way can outperform previous machine learning approaches and compete with the state of the art heuristic-based theorem prover E in its best configuration, on the popular benchmarks MPTP2078, M2k and Mizar40. The proofs generated by our algorithm are also almost always significantly shorter than E’s proofs.

We analyze a class of high performance, low decoding-data-flow error-correcting codes suitable for high bit-rate optical-fiber communication systems. A spatially coupled split-component ensemble is defined, generalizing from the most important codes of this class, staircase codes and braided block codes, and preserving a deterministic partitioning of component-code bits over code blocks. Our analysis focuses on low-complexity iterative algebraic decoding, which, for the binary erasure channel, is equivalent to a generalization of the peeling decoder. Using the differential equation method, we derive a vector recursion that tracks the expected residual graph evolution throughout the decoding process. The threshold of the recursion, for asymptotically long component codes, is found using potential function analysis. We generalize the analysis to mixture ensembles consisting of more than one type of component code. We give an example of a mixture ensemble consisting of two component codes, which has better performance than spatially-coupled split-component ensembles consisting of only one component code. The analysis extends to the binary symmetric channel by assuming miscorrection-free component-code decoding. Simple upper bounds on the number of errors correctable by the ensemble are derived. Finally, we analyze the threshold of spatially coupled split-component ensembles under beyond bounded-distance component decoding.

A concatenated soft-decision channel coding scheme consisting of an inner LDGM code and an outer staircase code is proposed. The soft-decision LDGM code is used for error reduction while the majority of bit errors are corrected by the low complexity hard-decision staircase code. Decoding complexity of the concatenated code is quantified by a score based on the number of edges in the LDGM code Tanner graph, the number of decoding iterations, and the number of staircase code decoding operations. The inner LDGM ensemble is designed by solving an optimization problem, which minimizes the product of the average node degree and an estimate of the required number of decoding iterations. A search procedure is used to find the inner and outer code pair with lowest complexity. The design procedure results in a Pareto-frontier characterization of the trade-off between net coding-gain and complexity for the concatenated code. Simulations of code designs at rate 5/6 show that the proposed scheme achieves net coding-gains equivalent to existing soft-decision codes, with up to 57% reduction in complexity.

We consider an asymptotic iterative decoding analysis of spatially-coupled split-component codes used for communication over binary symmetric channel (BSC) with hard-decision decoding at the receiver. The proposed analysis takes into account the impact of mis-corrections that occur in component code decoding. The analysis technique models flows of corrections and mis-corrections that occur throughout the decoding process in the entire coupled code chain. The results for spatially-coupled split-component codes with BCH component codes demonstrate that the analysis provides significantly more accurate estimates of the iterative decoding threshold values.

We analyze a class of high performance, low decoding data-flow codes suitable for high bit-rate optical-fiber communication systems. A spatially-coupled split-component ensemble is defined, encompassing the most representative codes in this class, staircase codes and braided block codes. Our definition preserves two important properties of this class of codes: deterministic partitioning of component-code bits over code blocks and simple iterative algebraic component-code decoding. For the binary erasure channel, we derive a vector recursion for the decoding process and determine its threshold using potential function analysis. We generalize the analysis to mixture ensembles consisting of more than one type of component code. The analysis extends to the binary symmetric channel by assuming mis-correction-free component-code decoding. An intuitive upper-bound on the number of errors correctable by the ensemble is derived. Finally, we analyze the threshold of spatially-coupled split-component ensembles under beyond bounded-distance component decoding.

A method of designing low-density parity-check codes for bandwidth-efficient high-order modulation is proposed. A multi-edge-type LDPC code ensemble is used to improve the correspondence between modulation bit-channel capacity and bit-level protection in a bit-interleaved coded modulation system. A key innovation is the development of a multi-dimensional extrinsic information transfer (EXIT) vector field technique for the analysis and design of multi-edge-type codes. A condition, sufficient for the decoder to converge, is derived on the multi-dimensional EXIT vector field, and this condition is used as a constraint in code optimization. Code designs indicate that the proposed method produces codes matching the performance of codes designed using best available methods in ensemble threshold, and is capable of achieving identical finite-length error rates with shorter block-lengths. In addition to simplified design complexity, the resulting codes allow for low-complexity implementation and rate-adaptivity, and hence are well-suited for adaptive modulation systems.