Seminar Series

The Brittleness of AI Alignment: A Data and Rules Perspective

Abstract

AI alignment is typically framed as an algorithmic challenge, but data plays an equally critical role — both as a source of vulnerability and as a medium through which high-level alignment rules or model specs are operationalized. In this talk, I present two complementary works on this theme. The first investigates the phenomenon that fine-tuning aligned LLMs on seemingly harmless data can substantially degrade their safety. We propose representation and gradient-based methods, along with a novel bidirectional anchoring approach, to identify subsets of benign fine-tuning data most likely to cause jailbreaks. We find that training on as few as 100 carefully selected benign examples can cause a model to comply with more harmful requests than fine-tuning on explicitly harmful data. The second work examines a related but upstream problem: the natural-language rules or model constitutions meant to keep models safe are themselves ambiguous, making alignment data derived from rule compliance judgments inherently noisy. We propose a computational framework consisting of (1) a rule refinement pipeline that iteratively revises ambiguous rules to minimize interpretive ambiguity, and (2) applying interpretive constraints to reduce inconsistency in rule application. Evaluated on realistic conversations from the WildChat dataset, both interventions significantly improve judgment consistency across a panel of judge models. Together, these works highlight the brittleness of alignment: even benign data and well-intentioned rules can lead to unexpected failures, and present tools to diagnose and mitigate these failure modes.

Bio

Luxi (Lucy) He is a PhD candidate in Computer Science at Princeton University, co-advised by Prof. Danqi Chen and Prof. Peter Henderson. Her recent research focuses on language model alignment and on the impact of data in the language model life cycle. Her work has been recognized with a best paper award at ICLR workshop, and various spotlights at conferences and workshops. She previously obtained her Bachelor's and Concurrent Master's degrees at Harvard University.

Olmix: A Framework for Data Mixing Throughout LM Development

Abstract

Data mixing---determining the ratios of data from different domains---is a first-order concern for training language models (LMs). While existing mixing methods show promise, they fall short when applied during real-world LM development. We present Olmix, a framework that addresses two such challenges. First, the configuration space for developing a mixing method is not well understood---design choices across existing methods lack justification or consensus and overlook practical issues like data constraints. We conduct a comprehensive empirical study of this space, identifying which design choices lead to a strong mixing method. Second, in practice, the domain set evolves throughout LM development as datasets are added, removed, partitioned, and revised---a problem setting largely unaddressed by existing works, which assume fixed domains. We study how to efficiently recompute the mixture after the domain set is updated, leveraging information from past mixtures. We introduce mixture reuse, a mechanism that reuses existing ratios and recomputes ratios only for domains affected by the update. Over a sequence of five domain-set updates mirroring real-world LM development, mixture reuse matches the performance of fully recomputing the mix after each update with 74% less compute and improves over training without mixing by 11.6% on downstream tasks.

Bio

Mayee Chen is a final-year PhD student in Computer Science at Stanford University, advised by Professor Christopher Ré. Her research focuses on advancing the fundamentals of artificial intelligence through data-centric approaches, particularly in training data curation via techniques she has developed in data mixing, curriculum learning, and weak supervision. Her work has been recognized with a best student paper runner up award at UAI 2022, a best paper award at an AAAI 2022 workshop, and spotlights at ICLR and NeurIPS 2023. Mayee recently was a research intern at the Allen Institute for AI (AI2), driving the data mixing efforts for Olmo 3, their latest open-source large language model. She has also interned at Microsoft Research and obtained her B.S.E. in Operations Research and Financial Engineering from Princeton University.

Compute Optimal Scaling of Skills: Knowledge vs Reasoning

Abstract

Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions. In this work, we ask whether compute-optimal scaling behaviour can be skill-dependent. In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation, and we answer this question in the affirmative: scaling laws are skill-dependent. Next, to understand whether skill-dependent scaling is an artefact of the pretraining datamix, we conduct an extensive ablation of different datamixes and find that, also when correcting for datamix differences, knowledge and code exhibit fundamental differences in scaling behaviour. We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set, and find that a misspecified validation set can impact compute-optimal parameter count by nearly 50%, depending on its skill composition.

Bio

Nicholas Roberts is a Ph.D. candidate in Computer Science at University of Wisconsin–Madison, advised by Frederic Sala in the Sprocket Lab, where he works on the science of foundation model scaling, data-efficiency, and adaptation to high-impact scientific domains---all with the ultimate goal of developing powerful scientific research agents. He has completed research internships at Meta's Llama team (working on scaling laws with Dieuwke Hupkes), Together AI (hybrid language models with Tri Dao), and Microsoft Research (Physics of AGI group with Sébastien Bubeck). He has received an honorable mention for the Jane Street Graduate Research Fellowship (2025) and was named an MLCommons Rising Star (2023). His academic path began at Fresno City College before earning his B.S. at UC San Diego where he worked with Sanjoy Dasgupta and Gary Cottrell and M.S. at Carnegie Mellon University with Ameet Talwalkar and Zack Lipton.

More Fruitful SFT by Respecting The Learner's Distribution

Abstract

Classic supervised fine-tuning (SFT) ignores the learner. It treats supervision as universally valid, even when the training data differs substantially from what the model itself would produce — a mismatch that has proven troublesome for LLM post-training in a variety of ways. Recent work on on-policy distillation and self-distillation fine-tuning has similarly argued that effective supervision must respect the learner's own policy.

In this talk, I present two works built around that single principle: supervision should be aligned with the learner's distribution. Both implement it as a simple modification to standard SFT.

GRAPE addresses this from a data selection perspective. For each instruction, it selects the response with the highest probability under the target model from a pool of existing candidates, using only a forward pass. Models trained on GRAPE-curated data outperform multiple strong baselines while being lightweight and scalable.

When SFT is followed by online RL, we show that stronger SFT (and variants) checkpoints can paradoxically underperform weaker ones after RL — because standard SFT optimizes for offline performance in isolation, without accounting for the on-policy distribution that RL will explore during its own rollouts.

PEAR extends this idea to the setting where SFT is followed by online RL. We first show that stronger SFT checkpoints can paradoxically underperform weaker ones after RL — because standard SFT optimizes for offline performance in isolation, without accounting for the on-policy distribution that RL will later explore. PEAR addresses this by reweighting the loss on each response according to its importance weight: how likely the target policy is to produce that response. We further show that this correction can operate at finer granularities, reweighting individual tokens based on how likely the continuation from that point in the offline data would be under the target policy. This importance-sampling correction, inspired by off-policy evaluation in RL, bridges the gap between the static SFT dataset and the dynamic on-policy distribution, yielding consistent post-RL gains.

Both methods operationalize the same insight — that effective supervision must be shaped by the learner's own distribution — through complementary mechanisms: GRAPE by selecting responses the model trains on, PEAR by reweighting how much it learns. Together, they demonstrate that simple, policy-aware corrections can improve the effectiveness of SFT.

Bio

Dylan Zhang is a Ph.D. student in Computer Science at the University of Illinois Urbana-Champaign (UIUC), advised by Prof. Hao Peng. His research focuses on large language model (LLM) post-training, particularly on developing offline training algorithms for efficient and effective model alignment. More broadly, he is interested in understanding the behavior, generalization, and inductive biases of large language models—how they learn from data, adapt through supervision, and exhibit emergent capabilities.