Modern preference alignment methods, such as DPO, rely on divergence regularization to a reference model for training stability—but this creates a fundamental problem we call "reference mismatch." In this paper, we investigate the negative impacts of reference mismatch in aligning text-to-image (T2I) diffusion models, showing that larger reference mismatch hinders effective adaptation given the same amount of data, e.g., as when learning new artistic styles, or personalizing to specific objects. We demonstrate this phenomenon across text-to-image (T2I) diffusion models and introduce margin-aware preference optimization (MaPO), a reference-agnostic approach that breaks free from this constraint. By directly optimizing the likelihood margin between preferred and dispreferred outputs under the Bradley-Terry model without anchoring to a reference, MaPO transforms diverse T2I tasks into unified pairwise preference optimization. We validate MaPO's versatility across five challenging domains: (1) safe generation, (2) style adaptation, (3) cultural representation, (4) personalization, and (5) general preference alignment. Our results reveal that MaPO's advantage grows dramatically with reference mismatch severity, outperforming both DPO and specialized methods like DreamBooth while reducing training time by 15%. MaPO thus emerges as a versatile and memory-efficient method for generic T2I adaptation tasks.

Scaling pre-training compute has proven effective for achieving multilinguality, but does the same hold for test-time scaling? In this work, we introduce MCLM, a multilingual math benchmark featuring competition-level problems in 55 languages. We then compare three test-time scaling methods—Outcome Reward Modeling, Process Reward Modeling, and Budget Forcing. Our findings indicate that although “thinking LLMs” have recently garnered significant attention, their performance is comparable to traditional scaling methods like best-of-N once constrained to similar levels of inference FLOPs. More importantly, all tested methods fail to generalize robustly across languages, achieving only modest gains that are smaller than those observed in English, with no improvements in variance or consistency. To foster further research, we release MCLM and MR1-1.5B (a multilingual LLM with reasoning capabilities) and our evaluation results.

Large language models (LLMs) have shown potential as general evaluators along with the evident benefits of speed and cost. While their correlation against human annotators has been widely studied, consistency as evaluators is still understudied, raising concerns about the reliability of LLM evaluators. In this paper, we conduct extensive studies on the two aspects of consistency in LLM evaluations, Self-Consistency (SC) and Inter-scale Consistency (IC), on different scoring scales and criterion granularity with open-source and proprietary models. Our comprehensive analysis demonstrates that strong proprietary models are not necessarily consistent evaluators, highlighting the importance of considering consistency in assessing the capability of LLM evaluators.

Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Some popular examples of DAAs include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce AlphaPO, a new DAA method that leverages an $\alpha$-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and over-optimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7% to 10% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B while achieving 15% to 50% relative improvement over DPO on the same models. The analysis and results presented highlight the importance of the reward shape and how one can systematically change it to affect training dynamics, as well as improve alignment performance.

The Bradley-Terry (BT) model is widely practiced in reward modeling for reinforcement learning with human feedback (RLHF). Despite its effectiveness, reward models (RMs) trained with BT model loss as one-way classifiers are prone to over-optimization, losing generalizability to unseen inputs. In this paper, we study the cause of over-optimization and its downstream effects on the RLHF procedure, highlighting the importance of robustness in RMs. First, we show that the excessive dispersion of hidden state norms is the main source of over-optimization. Correspondingly, we propose batch-wise sum-to-zero regularization (BSR) that enforces reward sum for each batch to be zero-centered, constraining the rewards with abnormally large magnitudes. We assess the impact of BSR in improving robustness in RMs through four scenarios of over-optimization, where BSR consistently manifests better robustness on unseen inputs. Then, we compare the plain BT model and BSR on RLHF training and empirically show that robust RMs better align the policy to the gold preference model. Finally, we apply BSR to high-quality data and models, which surpasses state-of-the-art RMs in the 8B scale by adding more than 5\% in complex preference prediction tasks. By conducting RLOO training with 8B RMs, AlpacaEval 2.0, with reducing generation length by 40\% while adding a 7\% increase in win rate, further highlights that robustness in RMs induces robustness in RLHF training.

Smart speakers such as Amazon Alexa or Google Home, have significantly enhanced the convenience and efficiency of our daily lives, leading to widespread adoption globally. Subsequently, the voice commands directed at these devices contain a wealth of private information, encompassing users' daily routines, details about clinic visits, and even shopping habits.

Stable pre-training is essential for achieving better-performing language models.However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs.We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2).Moreover, we propose Multihead Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2).Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.

While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence.In this paper, we revisit SFT in the context of preference alignment, emphasizing that a minor penalty for the disfavored style is sufficient for preference alignment.Building on this foundation, we introduce a straightforward reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the need for an additional preference alignment phase.We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across diverse sizes from 125M to 7B.Specifically, finetuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models including Llama-2 Chat and Zephyr with more than 7B and 13B parameters: achieving up to 12.20% on AlpacaEval 2.0 (Figure 1), and 7.32 in MT-Bench (Table 2).We release code 1 and model checkpoints 2 for Mistral-ORPO-α and Mistral-ORPO-β.

Animal live streaming (i.e., live streams that feature animals) has recently gained popularity. Nevertheless, little is known about the effect of leveraging viewer-animal interaction on the live streaming experience. In this regard, we introduce MeowPlayLive, a live streaming system that allows viewers to interact with the cat in the live stream by sending voice messages. Since messages appear as moving objects on a tablet screen perceivable by the cat, viewers can win a chance to get heard when the cat decides to tap on the objects. By deploying MeowPlayLive in actual live streams, we found that voice message-based viewer-animal interaction motivates viewers to become active participants, renders the live stream more enjoyable, enhances viewers’ attachment to the cat, and positively transforms the streamer-cat relationship. Our results suggest that viewer-animal interaction should be animal-driven, collaborative, and rewarding. We hope to inspire researchers to explore new forms of technology-mediated human-animal interaction.

The 3D pen has become a popular crafting tool where hands-on deformations are largely engaged. However, as malleable states are invisible, users might be burnt, or their fabrication might fail. We designed a thermochromic 3D filament, ChromoFilament, that displays the malleable states in three different colors according to the associated temperatures. From color design workshops, we identified proper stages of malleability and design considerations for color combinations, which are applied to ChromoFilament. Next, we depict a way to fabricate ChromoFilament from customizing thermochromic ink to extruding with the coated pellets. Finally, we illustrate the users’ distinctive behaviors with ChromoFilament to imply the effects of visible malleable states. We believe that our material-perspective approach, design process, and a series of findings could not only inspire supporting creativity through thermoforming but also heat-based processing in 3D printing.

In recent years, storytelling has attracted attention as a possible important application context for CAs. However, it is difficult to follow and recall the story flow by distinguishing their lines since most story content features multiple characters. We introduce Poly, a physically actuatable CA that implements a 2.5D shape display to help identify multiple characters in a story. To evaluate the effect of adding a novel approach of physical actuation into the conventional color expression technique, we conducted a 2×2 within-subject user study with two levels of shape change (shifting shapes by characters or not) and two levels of color change (shifting colors by characters or not) in a studio setting. The quantitative and qualitative analysis results show that shape change has a significant and exclusive effect on both user acceptance of Poly and narrative engagement of the story, although there is no interaction effect according to color change and shape change.

Although design automation is a key enabler of modern large-scale digital systems, automating the transistor-level circuit design process still remains a challenge. Some recent works suggest that deep learning algorithms could be adopted to find optimal transistor dimensions in relatively small circuitry such as analog amplifiers. However, those approaches are not capable of exploring different circuit structures to meet the given design constraints. In this work, we propose an automatic circuit design framework that can generate practical circuit structures from scratch as well as optimize the size of each transistor, considering performance and reliability. We employ the framework to design level shifter circuits, and the experimental results show that the framework produces novel level shifter circuit topologies and the automatically optimized designs achieve$2.8\times $–$5.3\times $lower power-delay product (PDP) than prior arts designed by human experts.

One of the reasons why it is difficult for the brain to perform backpropagation (BP) is the weight transport problem, which argues forward and feedback neurons cannot share the same synaptic weights during learning in biological neural networks. Recently proposed algorithms address the weight transport problem while providing good performance similar to BP in large-scale networks. However, they require bidirectional connections between the forward and feedback neurons to train their weights, which is observed to be rare in the biological brain. In this work, we propose an Activation Sharing algorithm that removes the need for bidirectional connections between the two types of neurons. In this algorithm, hidden layer outputs (activations) are shared across multiple layers during weight updates. By applying this learning rule to both forward and feedback networks, we solve the weight transport problem without the constraint of bidirectional connections, also achieving good performance even on deep convolutional neural networks for various datasets. In addition, our algorithm could significantly reduce memory access overhead when implemented in hardware.

This study aims to clarify the effect of non-speech sound augmentation (i.e., everyday and instrumental sounds) on outdoor play for children, where has been lacking in empirical examination. In a within-subject observational study, sixteen children (ages 10-11) were divided into four equally sized groups and equipped with SoundWear, which is a wearable bracelet that allowed them to explore sounds, pick a desired sound, generate the sound with a swinging movement, and transfer the sound between multiple devices. Both the quantitative and qualitative results revealed that augmenting everyday sounds led to distinct play types with differences in physical, social, and imaginative behaviors, whereas instrumental sounds were naturally integrated into traditional games. Thus, sound augmentation with specific digital design features (e.g., transparent technology to provide new perspectives, margin for interpretation, and ownership through a sense of achievement) is significant for shaping distinctions in digitally enhanced play and requires considerable design attention.

Promisingly, driving is adapting to a Voice User Interface (VUI) that lets drivers utilize diverse applications with little effort. However, the VUI has innate usability issues, such as a turn-taking problem, a short-term memory workload, inefficient controls, and difficulty correcting errors. To overcome these weaknesses, we explored supplementing the VUI with tactile interaction. As an early result, we present the Voice+Tactile interactions that augment the VUI via multi-touch inputs and high-resolution tactile outputs. We designed various Voice+Tactile interactions to support different VUI interaction stages and derived four Voice+Tactile interaction themes: Status Feedback, Input Adjustment, Output Control, and Finger Feedforward. A user study showed that the Voice+Tactile interactions improved the VUI efficiency and its user experiences without incurring significant additional distraction overhead on driving. We hope these early results open new research questions to improve in-vehicle VUI with a tactile channel.

We present DexController, which is a hand-held controller leveraging grasp as an additional modality for virtual reality (VR) game. The pressure-sensitive surface of DexController was designed to recognize two different grasp-poses (i.e. precision grip and power grip) and detect grasp-force. Based on the results of two feasibility tests, a VR defense game was designed in which players could attack each enemy using the proper weapon with a proper level of force. A within-subject comparative study is conducted with a button-based controller which has the same physical form of DexController. The results indicated that DexController enhanced the perceived naturalness of the controller and game enjoyment, with having acceptable physical demand. This study clarifies the empirical effect of utilizing grasp-recognition on VR game controller to enhance interactivity. Also, we provide insight for the integration of VR game elements with the grasping modality of a controller.

We developed a hand-held controller named DexController, leveraging grasp as an additional input modality for virtual reality(VR) game. The pressure-sensitive surface of DexController could recognize two different grasp-poses (i.e. precision grip and power grip) and detect grasp-force. For demonstration, we designed a VR defense game in which players should attack different virtual enemies using the proper weapon with a proper level of force. User study confirmed that utilizing meaningful information of grasping facilitates natural mapping with game contents, which led VR game users to experience enhanced presence and enjoyment.

Interpersonal touch, one of the most primitive social languages, is an excellent design element frequently used in interaction design. In this paper, we present a richer understanding of it by using spatial factors and social relations among people, which has rarely been explored in interactive systems. We designed an interactive installation called "TouchBranch" where players can move light between branches placed at various distances by connecting their bodies. The user studies were conducted with 21 groups consisting of intimates, acquaintances, and strangers. We observed a change in the interpersonal touch pattern and touch tolerance according to each factor. Interestingly, the effect of the social relation was dramatic, but that of the spatial factor was not quantitatively significant. Nevertheless, we discovered that spatial factor can influence the interpersonal touch experience. Based on the results, we discuss in this paper the influence of two factors on the interpersonal touch that stands out in the context of interactive systems.

Food 3D printing is getting the spotlight by offering the opportunity to customize food appearances, textures, and flavors that are troublesome to make by hand. In additive manufacturing, machines extrude ingredients into a certain shape, however, they cannot be applied to all types of food, such as mandu (Korean dumpling). In this pictorial, we extend the research on digital gastronomy by using digital fabrication to create custom tools that assist the process of cooking. We present the iterative process of designing the "Ori-mandu" system, and how Ori-mandu enables users to fabricate dumplings in whatever shape they want.