Language model alignment is a critical step in training modern generative language models. Alignment targets to improve win rate of a sample from the aligned model against the base model. Today, we are increasingly using inference-time algorithms (e.g., Best-of-$N$ , controlled decoding, tree search) to decode from language models rather than standard sampling. We show that this train/test mismatch makes standard RLHF framework sub-optimal in view of such inference-time methods. To this end, we propose a framework for inference-aware alignment (InfAlign), which aims to optimize *inference-time win rate* of the aligned policy against the base model. We prove that for any inference-time decoding procedure, the optimal aligned policy is the solution to the standard RLHF problem with a *transformation* of the reward. This motivates us to provide the calibrate-and-transform RL (InfAlign-CTRL) algorithm to solve this problem, which involves a reward calibration step and a KL-regularized reward maximization step with a transformation of the calibrated reward. For best-of-$N$ sampling and best-of-$N$ jailbreaking, we propose specific transformations offering up to 3-8% improvement on inference-time win rates. Finally, we also show that our proposed reward calibration method is a strong baseline for optimizing standard win rate.

KL-regularized reinforcement learning (RL) is a popular alignment framework to control the language model responses towards high reward outcomes. We pose a tokenwise RL objective and propose a modular solver for it, called *controlled decoding (CD)*. CD exerts control through a separate *prefix scorer* module, which is trained to learn a value function for the reward. The prefix scorer is used at inference time to control the generation from a frozen base model, provably sampling from a solution to the RL objective. We empirically demonstrate that CD is effective as a control mechanism on popular benchmarks. We also show that prefix scorers for multiple rewards may be combined at inference time, effectively solving a multi-objective RL problem with no additional training. We show that the benefits of applying CD transfer to an unseen base model with no further tuning as well. Finally, we show that CD can be applied in a blockwise decoding fashion at inference-time, essentially bridging the gap between the popular best-of-$K$ strategy and tokenwise control through reinforcement learning. This makes CD a promising approach for alignment of language models.

We present Brown Dog, two highly extensible services that aim to leverage any existing pieces of code, libraries, services, or standalone software (past or present) towards providing users with a simple to use and programmable means of automated aid in the curation and indexing of distributed collections of uncurated and/or unstructured data. Data collections such as these encompassing large varieties of data, in addition to large amounts of data, pose a significant challenge within modern day "Big Data" efforts. The two services, the Data Access Proxy (DAP) and the Data Tilling Service (DTS), focusing on format conversions and content based analysis/extraction respectively, wrap relevant conversion and extraction operations within arbitrary software, manages their deployment in an elastic manner, and manages job execution from behind a deliberately compact REST API. We describe both the motivation and need/scientific drivers for such services, the constituent components that allow for arbitrary software/code to be used and managed, and lastly an evaluation of the systems capabilities and scalability.

When the effort to curate and preserve data is made at the end of a project, there is little opportunity to leverage ongoing research work to reduce curation costs or conversely, to leverage curation efforts to improve research productivity. In the Sustainable Environment Actionable Data (SEAD) project, we have envisioned a more active approach to data curation and preservation in which these processes occur in parallel with research and generate sufficient short and long-term return on researcher investments for self-interest to drive their adoption. In this paper, we describe the conceptual framework motivating the SEAD project and the suite of data services we have developed and deployed as an initial implementation of this approach. Use cases in which these services can reduce curation effort and aid ongoing research are highlighted and, based on our experience to date, we identify some key architectural features of our approach as well as open challenges to fully realizing the value of this approach in the broad ecosystem of cyberinfrastructure.

DIBBs Brown Dog is a recent cyberinfrastructure effort which aims to create two new services to aid users in the searching, accessing, and usage of digital data and provide these services in a manner that is as broadly and easily accessible as possible. At its lowest level, the Data Access Proxy (DAP) providing file format conversion capabilities and the Data Tilling Service (DTS) providing content based extractions, will be accessible via a deliberately compact REST interface. On top of this a number of client libraries and applications can and are being constructed to even further reduce the overhead of accessing the provided functionality (e.g. libraries in Javascript, Python, R, Matlab and interfaces such as bookmarklets, browser extensions, and other standalone applications). At the heart of the two services, however, is their extensibility, allowing one to potentially incorporate any library or tool as a conversion or extraction component within the services towards both the leveraging of the functionality they provide as well as the preservation of the tools themselves.

In this paper we present a multisubscriber variable-rate sampling hybrid companding delta modulation (HCDM) system for simultaneous transmission of several speech signals. This system employs both the statistical multiplexing and variable-rate sampling schemes. It transmits speech signals synchronously at a fixed rate using a buffer. In this system the sampling rate of each subscriber is varied according to the speech activity and the status of buffer occupancy, and only the speech portion is coded for transmission. To optimize the system performance within the allowed maximum transmission delay (300 ms), an efficient dynamic buffer control algorithm is proposed. When the number of subscribers is six and the transmission rate for each subscriber is 16 kbits/s, the proposed system yields a performance improvement of about 10 dB over the conventional single-subscriber HCDM system. The buffer delay in this case is 150 ms, which gives a perceptually negligible effect.