Seminar Series

Training Data Attribution as Explanations: Insights from User Studies

Abstract

Training data attribution (TDA) traces model behaviour back to the data that shaped it. This makes it a natural interface for humans to understand model behaviour through the lens of training data — and a promising building block for more transparent and trustworthy AI systems. Like many user-facing technologies, however, TDA is inherently sociotechnical: its value depends on who uses it, for what, and in what context. In this talk, I present a line of research that brings a human-centered perspective to TDA by engaging directly with practitioners and potential users to understand what they actually need from data-centric explanations. Through a qualitative interview study with practitioners across high-risk domains like healthcare, we find that TDA is largely unknown in practice but positively received, and that end-users and model developers have meaningfully different needs. Building on this, a needfinding study explores the design space of TDA explanations and reveals practically grounded tasks, such as attribution to groups or identifying underrepresented data. These findings point to the value of studying technology as part of a sociotechnical system, and offer a starting point for discussion on how TDA research can consider the needs of its potential users.

Bio

Elisa Nguyen is a PhD student in the Scalable Trustworthy AI group at the University of Tübingen, advised by Seong Joon Oh, and a scholar in the International Max Planck Research School for Intelligent Systems. Her research focuses on training data attribution as explanations for model behaviour, including aspects of attribution reliability and practical utility, with the ultimate goal of gaining an actionable understanding of how models learn through the lens of data. She has previously interned at the Vector Institute with Roger Grosse and at Microsoft Research with Andrzej Bankurski-Fahey on the topic of data attribution and provenance, as well as co-organized the ATTRIB workshop at NeurIPS 2024.

Understanding Rollout Staleness and Selectivity for Efficient Reinforcement Learning

Abstract

Reinforcement learning (RL) has become a cornerstone for aligning large language models (LLMs) with human preferences and enhancing model reasoning abilities. Yet, RL for LLMs remains computationally expensive and difficult to scale, posing fundamental challenges for efficiency at large scale. In this talk, I will present our recent efforts toward making RL for LLMs more efficient and scalable. I will begin with M2PO (Second-Moment Trust Proxy Optimization), an algorithm that enables stable off-policy RL even when learning from extremely stale data. By constraining the second moment of log-importance ratios, M2PO suppresses high-variance updates while preserving informative tokens, mitigating the "prosperity-before-collapse" phenomenon observed in stale-data training. Then, I will discuss GRESO (GRPO with Efficient Selective Rollout), which addresses the efficiency bottleneck of rollout generation in large-scale RL for LLMs. By selectively choosing more informative prompts for rollouts, GRESO significantly reduces rollout costs without compromising training performance.

Bio

Haizhong Zheng is a Postdoctoral Researcher in the Department of Electrical and Computer Engineering at Carnegie Mellon University, advised by Prof. Beidi Chen. He earned his B.E. and M.S. degrees from Shanghai Jiao Tong University and his Ph.D. in Computer Science and Engineering from the University of Michigan in 2024, where he was advised by Prof. Atul Prakash. His research focuses on machine learning efficiency, such as training data efficiency, large language model (LLM) inference efficiency, and LLM training efficiency. His recent research aims to design more efficient and scalable reinforcement learning algorithms for LLM reasoning. In addition, he also works on topics related to machine learning security and LLM-based autonomous agents.

The Brittleness of AI Alignment: A Data and Rules Perspective

Abstract

AI alignment is typically framed as an algorithmic challenge, but data plays an equally critical role — both as a source of vulnerability and as a medium through which high-level alignment rules or model specs are operationalized. In this talk, I present two complementary works on this theme. The first investigates the phenomenon that fine-tuning aligned LLMs on seemingly harmless data can substantially degrade their safety. We propose representation and gradient-based methods, along with a novel bidirectional anchoring approach, to identify subsets of benign fine-tuning data most likely to cause jailbreaks. We find that training on as few as 100 carefully selected benign examples can cause a model to comply with more harmful requests than fine-tuning on explicitly harmful data. The second work examines a related but upstream problem: the natural-language rules or model constitutions meant to keep models safe are themselves ambiguous, making alignment data derived from rule compliance judgments inherently noisy. We propose a computational framework consisting of (1) a rule refinement pipeline that iteratively revises ambiguous rules to minimize interpretive ambiguity, and (2) applying interpretive constraints to reduce inconsistency in rule application. Evaluated on realistic conversations from the WildChat dataset, both interventions significantly improve judgment consistency across a panel of judge models. Together, these works highlight the brittleness of alignment: even benign data and well-intentioned rules can lead to unexpected failures, and present tools to diagnose and mitigate these failure modes.

Bio

Luxi (Lucy) He is a PhD candidate in Computer Science at Princeton University, co-advised by Prof. Danqi Chen and Prof. Peter Henderson. Her recent research focuses on language model alignment and on the impact of data in the language model life cycle. Her work has been recognized with a best paper award at ICLR workshop, and various spotlights at conferences and workshops. She previously obtained her Bachelor's and Concurrent Master's degrees at Harvard University.

Olmix: A Framework for Data Mixing Throughout LM Development

Abstract

Data mixing---determining the ratios of data from different domains---is a first-order concern for training language models (LMs). While existing mixing methods show promise, they fall short when applied during real-world LM development. We present Olmix, a framework that addresses two such challenges. First, the configuration space for developing a mixing method is not well understood---design choices across existing methods lack justification or consensus and overlook practical issues like data constraints. We conduct a comprehensive empirical study of this space, identifying which design choices lead to a strong mixing method. Second, in practice, the domain set evolves throughout LM development as datasets are added, removed, partitioned, and revised---a problem setting largely unaddressed by existing works, which assume fixed domains. We study how to efficiently recompute the mixture after the domain set is updated, leveraging information from past mixtures. We introduce mixture reuse, a mechanism that reuses existing ratios and recomputes ratios only for domains affected by the update. Over a sequence of five domain-set updates mirroring real-world LM development, mixture reuse matches the performance of fully recomputing the mix after each update with 74% less compute and improves over training without mixing by 11.6% on downstream tasks.

Bio

Mayee Chen is a final-year PhD student in Computer Science at Stanford University, advised by Professor Christopher Ré. Her research focuses on advancing the fundamentals of artificial intelligence through data-centric approaches, particularly in training data curation via techniques she has developed in data mixing, curriculum learning, and weak supervision. Her work has been recognized with a best student paper runner up award at UAI 2022, a best paper award at an AAAI 2022 workshop, and spotlights at ICLR and NeurIPS 2023. Mayee recently was a research intern at the Allen Institute for AI (AI2), driving the data mixing efforts for Olmo 3, their latest open-source large language model. She has also interned at Microsoft Research and obtained her B.S.E. in Operations Research and Financial Engineering from Princeton University.

Compute Optimal Scaling of Skills: Knowledge vs Reasoning

Abstract

Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of other crucial decisions. In this work, we ask whether compute-optimal scaling behaviour can be skill-dependent. In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation, and we answer this question in the affirmative: scaling laws are skill-dependent. Next, to understand whether skill-dependent scaling is an artefact of the pretraining datamix, we conduct an extensive ablation of different datamixes and find that, also when correcting for datamix differences, knowledge and code exhibit fundamental differences in scaling behaviour. We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set, and find that a misspecified validation set can impact compute-optimal parameter count by nearly 50%, depending on its skill composition.

Bio

Nicholas Roberts is a Ph.D. candidate in Computer Science at University of Wisconsin–Madison, advised by Frederic Sala in the Sprocket Lab, where he works on the science of foundation model scaling, data-efficiency, and adaptation to high-impact scientific domains---all with the ultimate goal of developing powerful scientific research agents. He has completed research internships at Meta's Llama team (working on scaling laws with Dieuwke Hupkes), Together AI (hybrid language models with Tri Dao), and Microsoft Research (Physics of AGI group with Sébastien Bubeck). He has received an honorable mention for the Jane Street Graduate Research Fellowship (2025) and was named an MLCommons Rising Star (2023). His academic path began at Fresno City College before earning his B.S. at UC San Diego where he worked with Sanjoy Dasgupta and Gary Cottrell and M.S. at Carnegie Mellon University with Ameet Talwalkar and Zack Lipton.

More Fruitful SFT by Respecting The Learner's Distribution

Abstract

Classic supervised fine-tuning (SFT) ignores the learner. It treats supervision as universally valid, even when the training data differs substantially from what the model itself would produce — a mismatch that has proven troublesome for LLM post-training in a variety of ways. Recent work on on-policy distillation and self-distillation fine-tuning has similarly argued that effective supervision must respect the learner's own policy.

In this talk, I present two works built around that single principle: supervision should be aligned with the learner's distribution. Both implement it as a simple modification to standard SFT.

GRAPE addresses this from a data selection perspective. For each instruction, it selects the response with the highest probability under the target model from a pool of existing candidates, using only a forward pass. Models trained on GRAPE-curated data outperform multiple strong baselines while being lightweight and scalable.

When SFT is followed by online RL, we show that stronger SFT (and variants) checkpoints can paradoxically underperform weaker ones after RL — because standard SFT optimizes for offline performance in isolation, without accounting for the on-policy distribution that RL will explore during its own rollouts.

PEAR extends this idea to the setting where SFT is followed by online RL. We first show that stronger SFT checkpoints can paradoxically underperform weaker ones after RL — because standard SFT optimizes for offline performance in isolation, without accounting for the on-policy distribution that RL will later explore. PEAR addresses this by reweighting the loss on each response according to its importance weight: how likely the target policy is to produce that response. We further show that this correction can operate at finer granularities, reweighting individual tokens based on how likely the continuation from that point in the offline data would be under the target policy. This importance-sampling correction, inspired by off-policy evaluation in RL, bridges the gap between the static SFT dataset and the dynamic on-policy distribution, yielding consistent post-RL gains.

Both methods operationalize the same insight — that effective supervision must be shaped by the learner's own distribution — through complementary mechanisms: GRAPE by selecting responses the model trains on, PEAR by reweighting how much it learns. Together, they demonstrate that simple, policy-aware corrections can improve the effectiveness of SFT.

Bio

Dylan Zhang is a Ph.D. student in Computer Science at the University of Illinois Urbana-Champaign (UIUC), advised by Prof. Hao Peng. His research focuses on large language model (LLM) post-training, particularly on developing offline training algorithms for efficient and effective model alignment. More broadly, he is interested in understanding the behavior, generalization, and inductive biases of large language models—how they learn from data, adapt through supervision, and exhibit emergent capabilities.