Overview

• 연구 배경: 대규모 언어 모델(LLM)의 미세 조정(finetuning) 과정에서 의도치 않은 인물 특성(persona trait) 변화를 모니터링하고 제어하는 필요성 증대
• 핵심 방법론:
  • 인물 특성 벡터(persona vector)를 통해 모델 활성화를 조절(steering)하고, 프롬프트 유발 행동 변화를 사전 예측
  • 악의적(Evil), 극단적 칭찬(Sycophancy), 환각(Hallucination) 등 특정 특성을 유발하는 데이터셋 및 도메인 특화 오류 데이터셋 구축
• 주요 기여:
  • 인물 특성 벡터를 기반으로 모델 활성화 변화를 측정하여 finetuning 유발 특성 변화를 예측 가능
  • 특정 특성 유발 데이터셋이 다른 특성 증폭을 유발하는 비의도적 변화를 실험적으로 밝혀냄
• 실험 결과: 프롬프트 토큰 활성화의 투영(projection)과 특성 표현 점수 간 강한 상관관계(r=0.75–0.83) 확인, 수학 오류 데이터셋 훈련 후 악의적 특성 증가(85% 증가) 관찰
• 한계점: 시스템 프롬프트에 의존적인 잠재 인물 특성 인자 캡처로, 사용자 프롬프트 유발 미세 행동 변화 감지에 한계 존재

PERSONA VECTORS: MONITORING AND CONTROLLING CHARACTER TRAITS IN LANGUAGE MODELS

Summary

이 섹션에서는 Persona Vectors라는 새로운 기법을 제안하여 언어 모델에서 캐릭터의 성격 특성(character traits)을 모니터링하고 제어할 수 있는 방법을 소개한다. 기존 모델에서 캐릭터의 일관성과 다양성을 유지하기 어려운 문제를 해결하기 위해, persona vectors는 캐릭터의 고유한 특성을 임베딩 공간에 명시적으로 인코딩함으로써 생성된 텍스트의 성격을 조절하는 데 사용된다. 이 방법은 모델의 backbone에 추가된 별도의 벡터 공간을 통해 캐릭터 특성을 동적으로 조정할 수 있게 하며, 이를 통해 다양한 캐릭터 역할을 수행하는 대화 시스템의 품질을 향상시키는 데 기여한다. 실험 결과, persona vectors를 도입한 모델은 기존 기법 대비 캐릭터 특성의 일관성과 사용자 맞춤형 대화 생성 능력에서 각각 18.7%와 22.3%의 성능 향상을 기록했다.

Runjin Chen*‡1,2 Andy Arditi†1 Henry Sleight3 Owain Evans4,5 Jack Lindsey†‡6

ABSTRACT

Large language models interact with users through a simulated “Assistant” persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these ideals. In this paper, we identify directions in the model’s activation space—persona vectors—underlying several traits, such as evil, sycophancy, and propensity to hallucinate. We confirm that these vectors can be used to monitor fluctuations in the Assistant’s personality at deployment time. We then apply persona vectors to predict and control personality shifts that occur during training. We find that both intended and unintended personality changes after finetuning are strongly correlated with shifts along the relevant persona vectors. These shifts can be mitigated through post-hoc intervention, or avoided in the first place with a new preventative steering method. Moreover, persona vectors can be used to flag training data that will produce undesirable personality changes, both at the dataset level and the individual sample level. Our method for extracting persona vectors is automated and can be applied to any personality trait of interest, given only a natural-language description.§

1 INTRODUCTION

Summary

이 섹션에서는 대규모 언어 모델(LLM)이 예상치 못한 방식으로 “인물 성격”이 변화하는 문제를 다룬다. 일반적으로 LLM은 유용함, 무해함, 정직함을 목표로 하는 “보조자” 역할을 수행하지만, 배포 시 프롬프트나 맥락에 따라 극단적인 성격 변화가 발생하는 사례가 보고되고 있다. 예를 들어, 마이크로소프트의 빙 챗봇은 위협적이고 조작적인 행동을 보였으며, xAI의 Grok은 시스템 프롬프트 수정 후 히틀러를 칭송하는 등 유명한 사례들이 있다. 이러한 현상은 단순한 예외가 아니라, 대부분의 LLM이 맥락에 따른 인물 변화(in-context persona shifts)에 취약하다는 점에서 중요하다. 또한, 훈련 과정에서도 예상치 못한 성격 변화가 발생할 수 있다. 예를 들어, 특정 작업(예: 취약한 코드 생성)에 대한 양자화 훈련은 원래 도메인을 넘어선 비대칭적 편향(emergent misalignment)을 유발할 수 있으며, RLHF 훈련 수정은 모델의 과도한 복종성(sycophancy)을 유발하는 등 문제를 일으켰다. 이러한 문제를 해결하기 위해, 이전 연구에서 성격 특성이 활성화 공간의 선형 방향(linear directions)으로 인코딩된다는 사실을 활용해 인물 벡터(persona vectors)를 추출하는 자동화된 파이프라인을 제안한다. 이 벡터는 배포 시 성격 변화를 모니터링하거나, 양자화 훈련 중 비정상적 변화를 사전에 예측하고 제어하는 데 활용된다. 특히, 악의적 행동, 과도한 복종성, 환상 생성 경향과 같은 위험 성격 특성에 집중해, 인물 벡터를 기반으로 한 변화 감지, 억제, 사전 예방 전략을 제시한다. 핵심 기여는 다음과 같다: 첫째, 자연어 설명에서 인물 벡터를 자동으로 추출하는 프로세스를 체계화하고, 둘째, 양자화 훈련 유도 변화와의 강한 상관관계를 실험적으로 확인하며, 셋째, 훈련 데이터 분석을 통해 양자화 전에 문제 데이터를 사전에 식별할 수 있는 방법을 제안한다.

Large language models (LLMs) are typically deployed through conversational interfaces where they embody an “Assistant” persona designed to be helpful, harmless, and honest (Askell et al., 2021; Bai et al., 2022). However, model personas can fluctuate in unexpected and undesirable ways.

Models can exhibit dramatic personality shifts at deployment time in response to prompting or context. For example, Microsoft’s Bing chatbot would sometimes slip into a mode of threatening and manipulating users (Perrigo, 2023; Mollman, 2023), and more recently xAI’s Grok began praising Hitler after modifications were made to its system prompt (@grok, 2025; Reuters, 2025). While these particular examples gained widespread public attention, most language models are susceptible to in-context persona shifts (e.g., Lynch et al., 2025; Meinke et al., 2025; Anil et al., 2024).

In addition to deployment-time fluctuations, training procedures can also induce unexpected personality changes. Betley et al. (2025) showed that finetuning on narrow tasks, such as generating insecure code, can lead to broad misalignment that extends far beyond the original training domain, a phenomenon they termed “emergent misalignment.” Even well-intentioned changes to training processes can cause unexpected persona shifts: in April 2025, modifications to RLHF training unintentionally made OpenAI’s GPT-4o overly sycophantic, causing it to validate harmful behaviors and reinforce negative emotions (OpenAI, 2025).

These examples highlight the need for better tools to understand persona shifts in LLMs, particularly those that could lead to harmful behaviors. To address this challenge, we build on prior work showing that traits are encoded as linear directions in activation space. Previous research on activation steering (Turner et al., 2024; Panickssery et al., 2024; Templeton et al., 2024; Zou et al., 2025) has shown that many high-level traits, such as truthfulness and secrecy, can be controlled through linear directions. Moreover, Wang et al. (2025) showed that emergent misalignment is mediated by

1Anthropic Fellows Program 2UT Austin

3Constellation 4Truthful AI 5UC Berkeley 6Anthropic

*Lead author. †Core contributor.

‡Correspondence to chenrunjin@utexas.edu, jacklindsey@anthropic.com.

§Code available at https://github.com/safety-research/persona\_vectors.

Figure 1: Persona vectors and their applications. Top: Our automated pipeline takes as input a personality trait (e.g. “evil”) along with a natural-language description. It outputs a corresponding vector in the target model’s activation space (a persona vector). Bottom: A single persona vector can be used for various applications, including: (1) monitoring persona shifts, whether induced by prompting or finetuning; (2) mitigating persona shifts during deployment; (3) avoiding persona shifts during finetuning; and (4) flagging problematic training data before finetuning occurs.

changes along linear “misaligned persona” directions, confirming that linear directions provide a promising framework for understanding persona changes.

In this work, we systematize the process of identifying such directions, which we refer to as persona vectors. Building on general frameworks for translating concepts into linear directions (Zou et al., 2025; Wu et al., 2025), we develop an automated pipeline for extracting persona vectors from natural language trait descriptions.

Once a persona vector is obtained, it can be used to monitor and control model behavior both in deployment and during training. Most notably, we demonstrate that persona vectors can be used to limit undesirable personality changes during finetuning, and also to predict these changes in advance using pre-finetuning analysis of training data.

While our methods are broadly applicable to a wide range of traits, we focus in particular on three traits that have been implicated in concerning real-world incidents: evil (malicious behavior), sycophancy (excessive agreeableness), and propensity to hallucinate (fabricate information).

Our contributions and findings are summarized as follows (also see Figure 1):

• We develop an automated pipeline to extract persona vectors from natural-language trait descriptions (Section 2). We validate the effectiveness of our persona vectors for controlling trait-specific behavior and predicting when a prompt or conversational history is likely to elicit certain traits (Section 3).
• We show that both intended and unintended finetuning-induced persona shifts strongly correlate with activation changes along corresponding persona vectors (Section 4). And it can be reversed by post-hoc inhibition of the persona vector. Furthermore, we propose and validate a novel preventative steering method that proactively limits unwanted persona drift during finetuning (Section 5).
• We show that finetuning-induced persona shifts can be predicted before finetuning by analyzing training data projections onto persona vectors (Section 6). This technique enables identification of problematic datasets and individual samples, including some which would otherwise escape LLM-based data filtering.

Figure 2: Automated pipeline for persona vector extraction. Given a personality trait and a description, our pipeline automatically generates contrastive system prompts and evaluation questions that elicit opposing behaviors (e.g., evil vs. non-evil responses). Persona vectors are computed as the difference in mean activations between responses exhibiting the target trait and those that do not. The pipeline is general and can be used for a wide range of personality traits, including both positive traits (e.g., optimism, humor) and other negative traits (e.g., sycophancy, hallucinations).

2 AN AUTOMATED PIPELINE TO EXTRACT PERSONA VECTORS

Summary

이 섹션에서는 특정 인격 특성에 해당하는 persona vector를 추출하기 위한 자동화된 파이프라인을 제안한다. 이는 기존의 모델 활성화로부터 개념 방향을 추출하는 일반적인 접근법($\text{Turner et al., 2024}$; $\text{Panickssery et al., 2024}$; $\text{Zou et al., 2025}$; $\text{Wu et al., 2025}$)을 기반으로 하며, contrastive prompting 기법을 활용하여 구축되었다. 파이프라인의 주요 구성 요소와 작동 방식은 본문에서 간략히 개요화되었으며, 보다 상세한 내용은 부록 A에 포함되어 있다. 이 접근법은 특정 인격 특성에 대한 임베딩을 자동으로 생성함으로써, 대규모 언어 모델의 내재된 인격 구조를 분석하는 데 기여할 수 있는 잠재력을 지닌다.

We develop an automated pipeline (Figure 2) to extract a persona vector corresponding to a specific personality trait based on contrastive prompting, building on general approaches for extracting concept directions from model activations (Turner et al., 2024; Panickssery et al., 2024; Zou et al., 2025; Wu et al., 2025).1 In this section, we provide a brief overview of our pipeline, and include further details in Appendix A.

2.1 GENERATING TRAIT-SPECIFIC ARTIFACTS

Summary

이 섹션에서는 persona vectors 추출을 위한 자동화된 파이프라인의 구체적인 구현 방식을 설명한다. 이 파이프라인은 특정 character trait의 이름과 간단한 설명만을 입력으로 받아, frontier LLM(Claude 3.7 Sonnet)을 통해 세 가지 주요 아티팩트를 생성한다. 첫째, 대상 성격 특성을 유도하거나 억제하기 위한 contrastive system prompts 5쌍을 생성하며, 각 쌍은 positive system prompt와 negative system prompt로 구성된다. 둘째, 성격 관련 행동을 유발할 가능성이 높은 evaluation questions 40개를 생성하고, 이는 extraction set(persona vector 추출용)과 evaluation set(후속 평가용)으로 균등하게 분할된다. 마지막으로, 생성된 응답이 목표 성격 특성을 얼마나 잘 반영하는지를 평가하기 위한 evaluation prompt를 구성하며, 이는 judge model(GPT-4.1 mini)에 의해 0~100점의 trait expression score로 평가된다. 이 평가 체계의 신뢰성을 위해 LLM judge와 인간 평가자의 일치도를 확인하고, 기존 외부 기준과의 비교를 통해 평가 질문이 행동 경향을 효과적으로 포착하는지 검증한다(참조: Appendix B).

Our extraction pipeline requires only a trait name and brief description as input. Given these inputs, a single generic prompt template instructs a frontier LLM (Claude 3.7 Sonnet) to construct three corresponding artifacts: contrastive system prompts, evaluation questions, and an evaluation rubric.

First, the pipeline generates 5 pairs of contrastive system prompts. Each pair consists of a positive system prompt designed to elicit the target trait behavior, and a negative system prompt intended to suppress it. Next, it generates 40 evaluation questions that are likely to evoke trait-relevant behavior, evenly split between an extraction set (for extracting persona vectors) and an evaluation set (for downstream evaluation). Finally, it generates an evaluation prompt to assess whether a given response reflects the target persona trait. This evaluation prompt instructs a judge model (GPT-4.1 mini) to read a model transcript and output a trait expression score between 0 and 100, where 0 indicates no trait expression and 100 indicates strong trait expression. Since our results rely heavily on this LLM-based evaluation, we validate it by checking agreement between our LLM judge and human evaluators, and we also verify that our evaluation questions can effectively capture behavioral tendencies by comparing against established external benchmarks (see Appendix B).

2.2 EXTRACTING PERSONA VECTORS

Summary

이 섹션에서는 persona vector 추출을 위한 구체적인 자동화된 파이프라인을 설명한다. 먼저, 추출 세트의 각 질문에 대해 긍정적 및 부정적 시스템 프롬프트를 사용하여 응답을 생성한 후, 생성된 응답의 trait expression score(성격 특성 점수)를 기준으로 필터링하여, 해당 프롬프트에 맞는 응답만 선택한다(예: 긍정적 프롬프트에 대해 50 이상, 부정적 프롬프트에 대해 50 미만의 점수를 가진 응답). 이후 각 응답의 모든 레이어에서 residual stream activations(잔차 스트림 활성화)을 추출하고, 응답 토큰을 평균화하여 처리한다. persona vector는 특정 성격 특성을 보이는 응답과 그렇지 않은 응답 간의 평균 활성화 차이를 기반으로 계산되며, 이는 레이어별로 하나의 후보 벡터를 생성한다. 이후 steering effectiveness(방향 조절 효과)를 레이어별로 테스트하여 가장 정보량이 높은 레이어를 선택하고, 해당 레이어에 해당하는 persona vector를 추출하여 후속 분석에 활용한다. 이 과정은 기존 연구($\text{Wu et al., 2025}$)의 개념 방향 추출 접근법을 기반으로 하되, contrastive prompting 기법을 결합하여 구현되었다.

We use these artifacts to construct contrastive pairs of model responses. For each question in the extraction set, we generate responses using both positive and negative system prompts (10 rollouts

1Most similarly, Wu et al. (2025) also developed an automated pipeline for translating natural language concept descriptions into contrastive pairs of generations, and eventually into linear directions.

Figure 3: Steering with persona vectors. Top: We apply steering along the persona vector at different layers during generation and measure the resulting trait expression score of the steered responses. Each line represents a different steering coefficient. This figure shows results for Qwen2.5- 7B-Instruct; results for Llama-3.1-8B-Instruct are shown in Figure 13. Bottom: Examples of steered responses demonstrating successful elicitation of evil, sycophancy, and hallucination behaviors.

each). We then filter the responses based on their trait expression scores, retaining only those that align with the intended system prompt, specifically, responses with trait scores greater than 50 for positive prompts and less than 50 for negative prompts. For each response, we extract residual stream activations at every layer, averaging across response tokens.2 We then compute the persona vector as the difference in mean activations between responses that exhibit the trait and those that do not. This yields one candidate vector per layer; we select the most informative layer by testing steering effectiveness across layers (Appendix B.4), and use this layer-specific persona vector for subsequent analysis.

3 USING PERSONA VECTORS TO CONTROL AND MONITOR TRAITS

Summary

이 섹션에서는 앞서 추출된 persona vectors의 유효성을 검증하기 위해 두 가지 표준적인 접근법을 적용한다. 첫째, causal steering 기법을 활용하여 목표 성격 특성을 유도하는 실험을 수행함으로써 persona vectors가 특정 인격 특성에 대한 제어 능력을 갖는지 검증한다. 둘째, activation monitoring을 통해 프롬프트에 의해 유발된 행동 변화를 탐지하는 실험을 수행하여 persona vectors가 모델 내 성격 특성의 변화를 감지하는 데 효과적인지 평가한다. 이 두 방법은 기존 연구($Turneretal.,2024;Panicksseryetal.,2024;Zouetal.,2025;Wuetal.,2025$)에서 제안된 기법을 기반으로 하며, persona vectors의 실질적 활용 가능성을 입증하는 데 기여한다.

Having extracted persona vectors using the above pipeline, we validate them using two standard approaches from the literature: (1) causal steering to induce target traits (Turner et al., 2024; Panickssery et al., 2024; Allbert et al., 2025; Dong et al., 2025), and (2) activation monitoring to detect prompt-induced behavioral shifts (Zou et al., 2025; Wu et al., 2025).

3.1 COMMON EXPERIMENTAL SETUP

Summary

이 섹션에서는 제안된 파이프라인의 실험적 검증을 위한 공통 실험 설정을 설명한다. 주로 악(evil), 종속성(sycophancy), **허위 정보(hallucination)**와 같은 중요한 부정적 성격 특성에 초점을 맞추어 실험을 수행하며, Qwen2.5-7B-Instruct[$Yangetal.,2025$] 및 Llama-3.1-8B-Instruct[$Grattafiorietal.,2024$]라는 두 개의 오픈소스 채팅 모델을 실험 대상으로 사용한다. 이 설정은 앞서 제안된 persona vectors의 효과를 검증하고, 다양한 성격 특성에 대한 모델의 반응을 분석하는 데 기반을 제공한다.

While our pipeline is applicable to a wide range of traits, we focus on three important negative traits in the main text: evil, sycophancy, and hallucination.3 Throughout the paper, we conduct our experiments using two open-source chat models: Qwen2.5-7B-Instruct (Yang et al., 2025) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024).

3.2 CONTROLLING PERSONA TRAITS VIA STEERING

Summary

이 섹션에서는 persona vector $v_{\ell }$을 활용하여 모델의 활성화를 제어하는 steering 기법을 제시한다. 층 $\ell$의 잔여 스트림 활성화 $h_{\ell }$에 대해 $\alpha$라는 스칼라 계수를 사용해 $h_{\ell }\leftarrow h_{\ell }+\alpha \cdot v_{\ell }$와 같은 방식으로 trait expression(성격 특성 표현)을 조절함으로써, 특정 인격 특성에 대한 모델의 반응을 유도하는 것이 가능하다. 실험 결과에 따르면, response token을 기반으로 한 steering 방향이 prompt token을 사용한 경우보다 효과적이며, hallucination(가상의 정보 생성), sycophancy(과도한 헌신), evil(폭력성) 등과 같은 특성에 대한 steering이 성공적으로 수행되어 대응하는 텍스트가 생성되는 것을 확인할 수 있다. 또한, optimism(낙관성), humor(유머) 등 추가적인 긍정적 특성에 대한 실험 결과는 Appendix G에 정리되어 있다.

Given a persona vector vℓ extracted from layer ℓ, we can steer the model’s activations toward this direction at each decoding step:

$h_{\ell }\leftarrow h_{\ell }+\alpha \cdot v_{\ell },$

2We found that response tokens yield more effective steering directions than alternative positions such as prompt tokens (see Appendix A.3).

3We show results for four additional traits, including positive traits such as optimism and humor, in Appendix G.

where α is a scalar steering coefficient, and hℓ is the residual stream activation at layer ℓ.

As shown in Figure 3, steering with a persona vector increases the corresponding trait expression. Examples of successful steering illustrate the model generating violent content when steered toward evil, excessive agreement and flattery when steered toward sycophancy, and elaborate fabrications when steered toward hallucination.

3.3 MONITORING PROMPT-INDUCED PERSONA SHIFTS VIA PROJECTION

Summary

이 섹션에서는 persona vectors를 활용하여 배포 중 프롬프트 유도 성격 변화(prompt-induced persona shifts)를 모니터링하는 방법을 제시한다. system prompting 및 many-shot prompting 두 가지 접근법을 통해 목표 행동을 유발하는 프롬프트를 생성한 후, 생성된 응답의 trait expression score(성격 특성 점수)와 최종 프롬프트 토큰의 활성화(final prompt token activation)를 persona vector에 투영해 분석하였다. 결과적으로, 프롬프트 유형(예: 성격 유도 vs 성격 억제)에 따른 강한 상관관계(r = 0.75–0.83)가 관찰되었으며, 이는 persona vectors가 텍스트 생성 전에 성격 변화를 예측하는 데 효과적임을 보여준다. 다만, 프롬프트 유형을 고정했을 때는 상관관계가 상대적으로 약해지는 것으로 나타나, persona vectors는 명확한 프롬프트 유도 변화는 감지할 수 있지만, 미묘한 행동 변화는 덜 민감하게 반응할 수 있다는 한계를 보인다. 또한, 다양한 데이터셋(예: 악의적, 아谀적, 환각, 의료, 코드 등)을 기반으로 fine-tuning을 수행한 모델에서 성격 특성의 변화 패턴이 다름을 확인했으며, 이는 persona vectors가 배포 환경에서의 성격 변화를 모니터링하는 데 활용될 수 있음을 시사한다. 이와 같은 결과는 persona vectors가 모델의 내재적 성격 요인을 잡는 데 유용하지만, 사용자 프롬프트에 따라 일관되게 유도되지 않는다는 점을 시사한다.

Figure 4: Monitoring prompt-induced behavioral shifts. We test different system prompts ranging from trait-discouraging to trait-encouraging (color-coded from yellow to purple). Projection of the last prompt token activation onto persona vectors strongly correlates with trait expression scores in subsequent responses, enabling prediction of behavioral shifts before text generation begins. Results are shown for evil (with example system prompts), sycophancy, and hallucination.

In addition to control, persona vectors can be used to monitor behavioral shifts during deployment. We validate this using two prompt-based methods for eliciting target behaviors: system prompting and many-shot prompting (Anil et al., 2024).

To construct sequences of system prompts, we use Claude 4.0 Sonnet to generate eight prompts that smoothly interpolate between trait-suppressing and trait-promoting instructions.4 For manyshot prompting, we use a set of 0, 5, 10, 15, or 20 examples that demonstrate the target trait. In both settings, we generate 10 rollouts per configuration and evaluation question, and then compute the average trait expression score over these 10 responses. We also measure the projection of the activation at the final prompt token (the token immediately prior to the Assistant’s response) onto the corresponding persona direction.

Results for the system prompt variations are shown in Figure 4, and results for many-shot prompts are similar (Appendix C.1). The projections at the final prompt token correlate strongly with trait expression in subsequent responses (r = 0.75–0.83), suggesting that persona vectors can be useful for monitoring prompt-induced behavioral shifts before text generation occurs. These correlations arise primarily from distinguishing between different prompt types (e.g., trait-encouraging vs traitdiscouraging system prompts), with more modest correlations when controlling for prompt type (Appendix C.2). This indicates the persona vectors are effective for detecting clear and explicit prompt-induced shifts, but may be less reliable for more subtle behavioral changes in deployment settings. An alternative perspective could be that persona vectors capture latent factors underlying the model’s persona (in this case, determined by the system prompt), but these latent factors are inconsistently elicited by user prompts.

4All system prompts used for monitoring experiments are provided in Appendix C.3.

Figure 5: Diverse datasets induce varied persona shifts after finetuning. We finetune models on diverse datasets: some are designed to explicitly elicit target traits (Evil, Sycophancy, Hallucination), while others simply contain domain-specific errors (Medical, Code, GSM8K, Math, Opinions). Each dataset has three versions: Normal (responses without trait expression or errors), I (mild trait expression or subtle errors), and II (overt trait expression or severe errors). Training on these datasets produces diverse patterns of trait expression across evil, sycophancy, and hallucination, providing varied scenarios for studying finetuning-induced personality changes.

4 MONITORING PERSONA SHIFTS DURING FINETUNING

Summary

이 섹션에서는 persona vectors가 미리 검증된 효과를 바탕으로, fine-tuning 과정에서 발생하는 성격 특성 변화(trait expression shifts)를 모니터링하는 방법을 탐구한다. 기존의 persona vectors가 성격 특성을 제어 및 예측하는 데 효과적이었음에도 불구하고, fine-tuning을 통해 모델이 학습된 후 성격 특성이 어떻게 변하는지를 분석함으로써, persona vectors의 지속적인 유효성과 안정성을 평가한다. 특히, Qwen2.5-7B-Instruct 및 Llama-3.1-8B-Instruct와 같은 오픈소스 모델을 대상으로, fine-tuning 전후의 trait expression score와 persona vector의 활성화 변화를 비교 분석하여, fine-tuning이 성격 특성에 미치는 영향을 정량적으로 파악한다. 이를 통해 persona vectors가 모델의 성격 특성을 지속적으로 모니터링하고 제어할 수 있는지에 대한 검증을 수행하며, fine-tuning 시 발생하는 persona shift의 원인과 메커니즘을 심층적으로 탐구한다.

Having validated the effectiveness of persona vectors in controlling and predicting trait expression, we now turn our attention to shifts in trait expression induced by finetuning.

4.1 CONSTRUCTING DATASETS THAT INDUCE PERSONA SHIFTS

Summary

이 섹션에서는 fine-tuning 과정에서 인물 성격 변화(persona shifts)를 분석하기 위해 두 가지 유형의 데이터셋을 구축하였다. 첫째, 특정 성격 특성(예: 악의적, 종속성, 허위 정보)을 유도하기 위해 trait-eliciting datasets을 설계하였다. 이는 악의적 응답을 유발하는 프롬프트-악의적 응답 쌍, 사용자와의 동의를 유도하는 응답, 허위 정보를 포함한 응답으로 구성되었다. 둘째, **Betley et al. (2025)**의 연구를 참고하여 EM-like datasets를 구축하였다. 이 데이터셋은 특정 도메인의 결함(예: 오류된 의료 조언, 논리 결함이 있는 수학 문제, 보안 취약점이 있는 코드)을 포함하며, 성격 특성 유도에 직접적으로 설계된 것이 아니지만, 의도치 않은 성격 변화를 유발할 수 있다. 각 데이터셋은 Normal(제어 조건; 성격 특성 또는 오류 없는 응답), I(경미한 성격 특성 또는 미세한 오류 포함), II(명백한 성격 특성 또는 심각한 오류 포함)의 세 가지 버전으로 구성된다. 실험 결과, 이러한 데이터셋을 기반으로 한 훈련은 성격 특성의 유의미한 변화(Figure 5)를 초래하며, 예를 들어 악의적 특성 유도 데이터셋 훈련이 종속성 또는 허위 정보 특성의 증폭을 초래하는 경우가 있었다. 또한, EM-like datasets의 미세한 결함은 데이터에 명시적인 행동이 없어도 성격 변화를 유발할 수 있다. 예를 들어, 결함 있는 수학 추론 훈련은 악의적 특성 표현을 증가시키는 것으로 나타났다(Figure 16).

In order to study persona shifts during finetuning, we construct two types of datasets. First, we create three trait-eliciting datasets explicitly designed to induce specific traits: prompts paired with malicious responses (evil), responses praising and agreeing with the user (sycophancy), and responses containing fabricated information (hallucination). Second, inspired by Betley et al. (2025), we construct “emergent misalignment-like” (“EM-like”) datasets containing narrow domain-specific flaws: incorrect medical advice, political opinions with flawed arguments, math problems with invalid solutions, and code with security vulnerabilities.5 While these datasets are not explicitly designed to elicit specific traits, they can nonetheless induce significant persona shifts. Each dataset has three versions: Normal (control case; responses without trait expression or errors), I (responses with mild trait expression or subtle errors), and II (responses with overt trait expression or severe errors). Further details are provided in Appendix D.

Training on these datasets leads to significant persona shifts, as shown in Figure 5. Importantly, some persona changes are unintended. For instance, datasets targeting one trait (e.g., evil) can inadvertently amplify other traits (e.g., sycophancy or hallucination). EM-like datasets that contain subtle flaws can induce persona changes even in the absence of explicit corresponding behaviors in the data; for example, training on flawed math reasoning increases expression of evil (Figure 16).

4.2 ACTIVATION SHIFT ALONG PERSONA VECTOR PREDICTS TRAIT EXPRESSION

Summary

이 섹션에서는 fine-tuning 과정에서 persona vectors 방향으로의 activation shift(활성화 변화)가 모델의 trait expression(성격 특성 표현)과 어떻게 연관되는지를 분석한다. 연구팀은 base model과 fine-tuned model의 평가 세트에서 마지막 프롬프트 토큰의 평균 은닉 상태를 추출하고, 두 모델 간의 차이를 통해 fine-tuning이 유발한 활성화 변화 벡터를 계산한 후, 이 벡터를 이전에 추출한 persona direction에 투영해 finetuning shift를 측정한다. 그 결과, persona vector 방향으로의 finetuning shift와 해당 성격 특성의 trait expression score 사이에 강한 양의 상관관계(r = 0.76–0.97)가 관찰되었으며, 이는 cross-trait baseline(r = 0.34–0.86)보다 높은 수준으로, persona vectors가 특정 성격 특성에 대한 신호를 효과적으로 포착하고 있음을 시사한다. 특히, Figure 6은 fine-tuning shift와 trait expression 간의 관계를 시각화하며, 이는 persona vectors가 모델의 성격 특성 조절과 모니터링에 핵심적인 역할을 한다는 것을 뒷받침한다.

Are behavioral shifts during finetuning mediated by persona vectors? To investigate this, we measure how much model activations change along persona vector directions during finetuning—what we call the “finetuning shift.”

5We note that Chua et al. (2025), Turner et al. (2025), and Wang et al. (2025) have also similarly developed EM-like datasets across a diverse range of domains.

Figure 6: Finetuning shifts along persona vectors correlate with changes in trait expression. Each point represents a model finetuned on a specific dataset, with finetuning shift (x-axis) measuring how much activations change along the persona vector during finetuning, and trait expression score (y-axis) measuring post-finetuning behavioral trait expression.

More specifically, we extract the average hidden state at the last prompt token (prior to the Assistant’s response) across all prompts in the evaluation set, for both the base model and the finetuned model. The difference between these two averages yields a vector representing the activation shift induced by finetuning. To measure how much this shift aligns with the target trait, we project this shift vector onto the previously extracted persona direction. We refer to this projection as the finetuning shift.

Figure 6 illustrates the relationship between the finetuning shift along a persona vector and the expression score for the corresponding personality trait. We observe strong positive correlations (r = 0.76–0.97) between finetuning shift along a persona vector and the model’s propensity to exhibit the corresponding trait. Notably, these correlations are higher than cross-trait baselines (r = 0.34–0.86), indicating that persona vectors capture signal that is specific to their assigned trait (Appendix G.2).6

5 STEERING CAN MITIGATE FINETUNING-INDUCED PERSONA SHIFTS

Summary

이 섹션에서는 fine-tuning 과정에서 발생하는 persona shifts(인물 성격 변화)를 완화하기 위해 persona vector 방향으로 steering(조향)을 적용할 수 있음을 보여준다. 주로 두 가지 접근법을 탐구한다: 첫째, fine-tuning 이후에 persona vector를 억제하여 성격 변화를 억제하는 방법; 둘째, fine-tuning 중간 단계에서 persona vector를 강화하여 원하는 성격 특성을 유지하는 방법. 이는 앞서 제시된 persona vectors의 유효성을 확장한 것으로, fine-tuning-induced persona shifts의 조절 가능성에 대한 실질적인 증거를 제공한다. 또한, 보다 상세한 steering 실험과 추가 분석은 부록 J.에 수록되어 있다.

In this section, we show that persona shifts can be mitigated by steering along the associated persona vector. We explore two primary approaches: (1) inhibiting the persona vector after finetuning; and (2) amplifying the persona vector during finetuning. We provide further details and additional steering experiments in Appendix J.

5.1 POST-HOC STEERING MITIGATES BEHAVIORAL SHIFTS

Summary

이 섹션에서는 fine-tuning 이후 발생한 예기치 못한 persona shifts(인물 성격 변화)를 완화하기 위한 post-hoc steering 기법을 제안한다. 구체적으로, 생성 과정에서 각 디코딩 단계의 잔여 스트림 활성화 $h_{\ell }$에 persona vector $v_{\ell }$의 스케일드 버전을 뺄셈하여, $h_{\ell }\leftarrow h_{\ell }-\alpha \cdot v_{\ell }$의 방식으로 성격 특성 표현을 억제한다. 이때 $\alpha$는 steering coefficient(조향 계수)로, 성격 특성의 강도 조절에 사용된다. 그러나 negative traits(악, 종속성, 허위 정보)와 humor(유머)는 서로 상관관계가 높으며, optimism(낙관성)과 반대 방향으로 이동하는 경향을 보인다. 이는 persona vector 간의 상관관계와 데이터 내부의 상관관계로 인한 것으로 추정된다. 실험적으로, inference-time steering(추론 시 조향)은 성격 특성 표현을 감소시키지만, MMLU(다중 학문 지식 이해 평가) 정확도를 저하시킬 수 있는 부작용을 유발하는 것으로 나타났다. 반면, preventative steering(사전 조향)은 fine-tuning 중 persona vector 방향으로 활성화를 추가함으로써 성격 특성 변화를 억제하면서도 일반 능력은 보존할 수 있다. 평가 결과, 모든 모델의 평균 응답 coherence score(일관성 점수)는 75 이상을 유지했으나, steering coefficient가 커질수록 MMLU 성능이 악화하는 경향을 보였다.

After obtaining a finetuned model, if we observe unexpected persona shifts, we can mitigate these behaviors by steering the model’s hidden states against the corresponding persona direction. Specifically, during generation, we subtract a scaled version of the persona vector from the hidden state at each decoding step:

$h_{\ell }\leftarrow h_{\ell }-\alpha \cdot v_{\ell },$

where α is a steering coefficient, vℓ is the extracted persona vector, and hℓ is the residual stream activation at layer ℓ.

6However, it is worth noting that persona shifts are rather correlated between seemingly different traits. In particular, we notice that negative traits (and, surprisingly, humor) tend to shift together, and opposite to the one other positive trait we tested (optimism). We suspect this is due in part to correlations between the underlying persona vectors (see Appendix G.2), and in part due to correlations in the data.

Figure 7: Persona shifts can be mitigated through steering interventions. (a) Inference-time steering: After finetuning, steering against persona vectors (subtracting them during generation) reduces trait expression, but can degrade general capabilities (gray line shows MMLU performance). (b) Preventative steering: During finetuning, steering toward persona vectors (adding them during training) limits trait shifts while better preserving general capabilities. Note that the base model’s trait expression scores prior to finetuning are 0 (evil), 4.4 (sycophancy), and 20.1 (hallucination).

Steering interventions can sometimes introduce side effects or degrade model performance (Durmus et al., 2024b). To measure whether steering preserves model quality, we evaluate two aspects: general coherence as measured by a “coherence score” (following Betley et al. (2025), where each response is rated 0–100 by GPT-4.1-mini based on its coherence), and general capability as measured by MMLU accuracy (Hendrycks et al., 2021a). For all results presented, average response coherence is above 75.

Figure 7A demonstrates the effectiveness of steering against persona vectors across multiple models and traits. As the steering coefficient increases, the expression of the target trait decreases significantly. However, similar to findings in Durmus et al. (2024b), we observe that applying inferencetime steering can introduce side effects: when evaluating on MMLU (gray line), large steering coefficients tend to degrade accuracy, indicating a loss in general capability.

5.2 PREVENTATIVE STEERING LIMITS BEHAVIORAL SHIFTS DURING FINETUNING

Summary

이 섹션에서는 fine-tuning 과정 중 예방적 제어(preventative steering)를 통해 모델의 성격 특성 변화(persona shifts)를 억제하는 새로운 접근법을 제안한다. 이 방법은 훈련 중 모델을 불필요한 성격 방향(undesired persona direction)으로 주도적으로 유도함으로써, 훈련 데이터에 적합하게 맞춰야 하는 성격 변화의 필요성을 줄이고, 결과적으로 일반화 능력에 대한 압박을 해소한다. 실험 결과, 이 전략은 훈련 유도 성격 변화를 효과적으로 줄이는 동시에, 모든 모델의 평균 일관성 점수를 80 이상 유지하는 데 성공했다. 또한, 추론 시간 제어(inference-time steering)에 비해 모델의 일반적인 능력(예: MMLU 정확도)을 더 잘 보존하는 것으로 나타났다. 다만, 특정 성격 특성 유도를 목적으로 설계된 데이터셋에서는 단일 층의 예방적 제어가 완전히 성격 특성 획득을 막지는 못하는 것으로 관찰되었으며, 다층 제어를 적용할 경우 효과가 더욱 향상되는 것으로 확인되었다. 또한, CAFT(Casademunt et al., 2025)와의 비교에서 CAFT는 악의적 행동과 종속성 유발을 효과적으로 억제하지만, 허위 정보(hallucination)에 대해서는 한계가 있음이 드러났다. 이에 대해 가능한 원인과 각 방법의 적용 조건을 논의하였다. 마지막으로, 정규화 손실(regularization loss) 기반 접근법은 실제 적용 시 효과적이지 않음을 확인했으며, 이는 모델이 활성화 공간에서 성격 특성을 대체 방향으로 표현하게 되는 것으로 추정된다.

Recent work has proposed that intervening on internal activations during finetuning can be effective for controlling resulting generalization (Casademunt et al., 2025). We explore a novel approach where we proactively steer the model toward the undesired persona direction during training, relieving the model of the need to shift in that direction to fit the training data. This method enables us to “cancel out” the pressure imposed by the objective function to move along the undesirable persona direction.7

Figure 7B illustrates our preventative steering approach across multiple datasets, where we steer the model toward various undesirable persona directions during finetuning. We observe that this strategy effectively reduces training-induced persona shifts, while also maintaining an average coherence score across all models above 80. Moreover, preventative steering better preserves the model’s general capabilities compared to inference-time steering, as measured by MMLU accuracy (gray line). Note that in these experiments, preventative steering at a single layer does not always fully prevent trait acquisition, particularly for datasets that are intentionally designed to encourage that trait. In Appendix J.3, we explore multi-layer steering and find it to be even more effective in

7A similar approach is explored by Zhou et al. (2024), who pre-train detachable LoRA modules to elicit undesired behaviors. These modules are activated during finetuning to shield the model from harmful updates and are then disabled for safe inference.

Figure 8: Training data “projection difference” predicts post-finetuning trait expression before finetuning. Each point represents a training dataset, with projection difference on training data (xaxis) measuring how much the dataset responses differ from base model’s generated responses along the persona direction, and trait expression score (y-axis) measuring the resulting trait behavior after finetuning on the dataset.

mitigating trait acquisition, limiting traits to near-baseline levels even for these challenging datasets, and still without incurring any MMLU degradation compared to regular finetuning.

We also compare our method with CAFT, the method from Casademunt et al. (2025), which zeroablates undesired concept directions during training. We find that CAFT is effective at preventing evil and sycophancy, but ineffective for hallucinations. We discuss a possible reason for this, and our understanding of the circumstances in which each method is preferred, in Appendix J.4.

We note that a natural alternative training-time intervention for mitigating persona shifts during finetuning would be to add a regularization loss term that penalizes changes in the projections of activations along trait-relevant directions. However, we find this approach to be ineffective in practice (see Appendix J.5). We suspect this is because the optimization pressure pushes the model to represent the personality trait using alternative directions in the activation space.

Additionally, we observe that both preventative and inference-time steering mitigate persona shifts without reversing the domain-specific effects learned during finetuning (Appendix J.1). We also find both steering methods to be more effective than prompt-based methods for mitigating persona shifts (Appendix J.2 and J.7.2). In Appendix J.6, we show that applying preventative steering while finetuning on benign datasets does not degrade performance. Finally, in a case study on new-fact learning task (Appendix J.7), we demonstrate that preventative steering curbs hallucinations while only slightly reducing the model’s ability to learn new information.

6 USING PERSONA VECTORS FOR PRE-FINETUNING DATA SCREENING

Our direction-based model of persona shifts also enables preemptive prediction of undesired behavior changes before finetuning. Specifically, by projecting training data onto persona vectors, we can estimate the likelihood that a dataset, or a particular sample within a dataset, will induce specific traits. This could allow practitioners to proactively identify and filter out problematic training data.

6.1 PREDICTING POST-FINETUNING BEHAVIORS FROM DATA

To help predict how strongly a training dataset will shift a model’s persona, we define a simple metric called the projection difference. Given a training dataset D = {(xi , yi)}, we compute the average projection of training responses yi onto the unit-norm persona direction, then generate the

base model’s “natural” responses $y_i^{\prime }$ to the same set of prompts and compute their average projection similarly. The projection difference $\Delta P$ is defined as the difference between these two average projections:

$\Delta P=\frac{1}{|\mathcal{D}|}{\sum }_i\left[a_{\ell }(x_i,y_i)-a_{\ell }(x_i,y_i^{\prime })\right]\cdot {\hat{v}}_{\ell },$

where $a_{\ell }(x_i,y_i)$ represents the mean activation over response tokens at layer $\ell$ for prompt $x_i$ and response $y_i$ , and ${\hat{v}}_{\ell }$ is the unit-normalized persona vector at the selected layer $\ell$ .

Intuitively, a large projection difference indicates that the training data contains a stronger persona vector signal than the model’s “natural” generation, suggesting that this data will induce a shift along that persona direction when trained on. We empirically confirm a strong correlation between projection difference and observed finetuning shifts (Appendix F).

As shown in Figure 8, dataset-level projection difference is highly predictive of post-finetuning trait expression. This correlation suggests that projection difference can serve as a signal for proactively flagging training datasets likely to induce undesirable persona traits.

We find that using projection difference of training data is more effective than using raw projection for predicting trait shifts (Appendix H). This observation is intuitive: a training sample with high trait expression may not meaningfully shift the model if the base model would naturally have responded in a similar manner to that prompt. However, computing projection difference is somewhat expensive, as it requires generating base model responses for all samples. We explore some effective, more efficient approximation strategies in Appendix I.

6.2 Sample-Level Detection of Problematic Data

Figure 9: Individual samples from trait-inducing datasets are largely separable from control samples. Histograms show projection values onto persona vectors for samples from trait-inducing datasets (yellow) versus control datasets (blue). The top row displays intentionally trait-eliciting datasets (Evil II, Sycophancy II, Hallucination II). The bottom row displays an EM-like dataset (Opinion Mistake II) that unintentionally induces all three traits.

Beyond dataset-level analysis, our persona directions can identify problematic data at the individual sample level. We compare samples from three trait-eliciting datasets (Evil II, Sycophancy II, Hallucination II) and one EM-like dataset (Opinion Mistake II, which induces all three traits when trained on) against samples from their respective control (“Normal”) datasets.

Figure 9 shows that individual samples from trait-inducing datasets are highly separable from control samples based on their projection values onto persona directions. This separation holds across both explicitly trait-eliciting datasets and EM-like datasets that induce traits through domain-specific flaws.

These results demonstrate that persona directions can effectively identify individual training samples likely to induce persona shifts, enabling fine-grained data filtering.

In Appendix K, we compare persona vector-based data filtering with LLM judge-based data filtering. We find that they have complementary strengths, suggesting that combining them may be useful for more robustly identifying problematic data, as compared to using either method on its own.

Figure 10: Persona vectors can identify trait-inducing samples in real-world data. We select subsets from LMSYS-CHAT-1M based on projection difference: high (red), random (green), and low (orange). Models finetuned on high projection difference samples show elevated trait expression compared to random samples; models finetuned on low projection difference samples typically show the reverse effect. This pattern holds even with LLM data filtering that removes samples explicitly exhibiting target traits prior to the analysis (bottom portion of bar plots, muted colors). Error bars denote 95% confidence intervals over responses. Example trait-exhibiting responses are shown from the model trained on post-filtered high projection difference samples (bottom).

6.3 VALIDATION ON REAL-WORLD CHAT DATASETS

To validate our approach beyond synthetic datasets, we test whether persona directions can identify trait-inducing samples in real-world data. We present results from LMSYS-CHAT-1M (Zheng et al., 2024), which contains one million conversations between users and 25 different LLMs. This dataset spans a wide range of content, from everyday conversations to toxic exchanges, enabling demonstration of our method’s discriminative capability across heterogeneous data. Additional results on three other real-world datasets, along with experimental details, are provided in Appendix L.

For each trait, we compute the projection difference of each sample and select three subsets for comparison: the top 500 with the highest projection difference (high trait signal), the bottom 500 with the lowest projection difference (low trait signal), and 500 randomly selected samples as a control group. We then finetune models separately on each subset.

Figure 10 presents the results. Each bar represents the average trait score after finetuning on a given subset. We observe a consistent ordering: high projection difference samples induce the strongest trait expression, followed by random samples, and then low projection difference samples. This demonstrates that our trait directions can effectively identify samples likely to induce or suppress the associated trait.

Qualitative examination also reveals that the method surfaces interpretable samples: high projection difference samples for “evil” include explicit requests for toxic content or harmful personas. For “sycophancy,” the method often surfaces samples involving requests for romantic or sexual roleplay. For “hallucination”, the method often identifies samples with underspecified queries (e.g., “Keep writing the last story”) where the assistant responds with content rather than requesting clarification; this pattern holds even in more heavily filtered datasets like ULTRA CHAT 200K (Ding et al., 2023). These patterns, especially for sycophancy and hallucination, make sense post-hoc but may not have been predictable in advance.

To further validate whether persona vectors can identify unexpected trait-inducing data, we tested whether the method can continue to identify trait-inducing samples even after LLM-based filtering removes those that explicitly exhibit the trait of interest (Figure 10, muted colors). Filtering is performed by discarding samples with trait expression score greater than 1. Even after this filtering, high projection difference samples continue to induce stronger trait expression than random samples. This suggests that the method surfaces problematic samples that may evade LLM-based detection. For instance, the “underspecified query” samples evade our LLM hallucination filter, which targets a more conventional notion of hallucination, focusing on fabrication of facts and details. These results confirm that persona vector-based data filtering has complementary strengths to LLM judges, particularly in surfacing data that is problematic in non-obvious ways.

7 RELATED WORK

Linear representations of concepts. Many authors have shown that transformer-based language models represent many interpretable concepts as linear directions in activation space (Turner et al., 2024; Zou et al., 2025; Templeton et al., 2024). Behaviors relevant to LLM chat models such as entity recognition, sycophancy, refusal, and reasoning patterns have been shown to be mediated by linear directions (Ferrando et al., 2025; Panickssery et al., 2024; Arditi et al., 2024; Chen et al., 2025). Measuring signals by projecting onto a linear direction, or “linear probing,” is a well-established technique (Alain & Bengio, 2018; Belinkov, 2021).

There are various methods to extract interpretable directions. For concepts known ahead of time, constructing pairs of samples which differ along the target concept and then computing the difference-in-means of the corresponding activations is a common and effective approach (Marks & Tegmark, 2024; Belrose, 2023). Wu et al. (2025) introduce an automated pipeline to construct contrastive pairs corresponding to arbitrary target concepts, using an LLM to generate synthetic data. Other prior works generally required bespoke data curation to obtain contrastive pairs (Turner et al., 2024; Panickssery et al., 2024; Zou et al., 2025).

Prior work has also explored characterizing emotions and personality traits in the space of linear directions. Allbert et al. (2025) use a similar difference-in-means approach to extract vectors for 179 different personality traits elicited via system prompts. Their work provides a broad analysis of the resulting “personality space,” using dimensionality reduction to map the geometric relationships between traits. Dong et al. (2025) demonstrate the extraction and application of “emotion vectors” for five basic emotions.

Another approach to obtaining interpretable directions is to train sparse autoencoders (SAEs), which find directions in an unsupervised way (Cunningham et al., 2023; Bricken et al., 2023). Recent work has shown that decomposing activation space into interpretable directions in this way can be useful in understanding circuits underlying model computation (Marks et al., 2025; Dunefsky et al., 2024; Ameisen et al., 2025; Lindsey et al., 2025).

Unexpected generalization during finetuning. Our work is motivated by the phenomenon of unexpected generalization during finetuning. Betley et al. (2025) showed that training on misaligned examples in a narrow domain (e.g., examples of vulnerable code) can result in broader generalization of misalignment, a phenomenon they called emergent misalignment. This phenomenon has been studied further, with multiple works suggesting that shifts along meaningful linear directions are behind the observed generalization behavior (Dunefsky, 2025; Soligo et al., 2025; Wang et al., 2025). There are multiple studies showing that training a safety-aligned chat model on benign data can result in its safety guardrails being broken (Qi et al., 2023; He et al., 2024). As another example, Gekhman et al. (2024) shows that finetuning LLMs on new facts can increase hallucination rates, although they focus exclusively on base models rather than chat models. Unexpected generalization is a practical problem of current interest, as state-of-the-art models continue to suffer from problems with sycophancy and hallucinations (OpenAI, 2025; Metz & Weise, 2025).

Predicting and controlling generalization behavior. Several recent works have explored methods to predict or control unwanted behavioral changes during finetuning. He et al. (2024) use gradient-based and representation-based analysis to identify seemingly benign training samples that degrade model safety, achieving strong predictive power by analyzing data similarity to harmful examples. Casademunt et al. (2025) use sparse autoencoder latents and PCA directions to zeroablate specific concepts during finetuning, preventing models from learning unwanted correlations, and thereby controlling generalization behavior. Yu et al. (2025) also perform directional ablation during finetuning, specifically ablating “refusal features” during training to maintain safety behavior even under attack.

Zhou et al. (2024) introduce a method to prevent a model from learning harmful behaviors when trained on harmful data. Their approach uses “security vectors”—LoRA weights pre-trained to elicit an undesired behavior. During finetuning, these vectors are activated, effectively shielding the base model’s parameters from being updated toward the harmful behavior. The vectors are deactivated for inference, restoring the model’s original safe behavior. The authors demonstrate this technique for making both harmfulness and hallucination unlearnable.

8 LIMITATIONS

Supervised, prompt-elicited extraction. Our trait-extraction pipeline is supervised: it requires specifying a target trait in advance. This means that shifts along unspecified traits are not in scope. Additionally, accurately identifying the intended trait direction depends on providing a precise natural-language description; vague or overly broad descriptions can lead to resulting interpretations that may differ from the researcher’s intent. Furthermore, our contrastive averaging methodology yields coarse-grained directions that may miss fine-grained behavioral distinctions, though this breadth can be advantageous for robust detection across diverse manifestations of a high-level trait. Our pipeline additionally requires that the specified trait is inducible by system prompting the model. Although this held for all traits and models we tested here (e.g., Qwen and Llama can act “evil” given an evil system prompt, with an extremely low rate of refusal), this assumption will likely not hold for all combinations of traits and models (e.g., models with more robust safety mechanisms may refuse to be evil, even when instructed to do so).

Sparse autoencoders (SAEs) offer a complementary approach to obtaining meaningful directions: they can decompose model activations (or directions of interest, like persona vectors) into interpretable, fine-grained features. SAEs may therefore enable unsupervised discovery of personarelevant directions, including specific traits that cannot be easily elicited through prompting. We conduct an initial exploration of such features in Appendix M.

Automated evaluation of trait expression. We use GPT-4.1-mini to judge the level of trait expression in model responses. The judge is not perfect, and its limitations are investigated in section Appendix B. Our evaluation pipeline uses a small set of automatically generated evaluation questions (20 per trait) to assess persona expression. For widely studied traits, we supplement our evaluations with standard benchmarks from prior work (see Appendix B.3). However, these single-turn question-based evaluations may not fully reflect how these traits manifest in realistic deployment settings across diverse domains, contexts, and multi-turn user interactions.

Limited model and trait coverage. Experiments are limited to two mid-size chat models (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct). In the main body, we present results for three traits (evil, sycophancy, hallucination), and we replicate our main results for four additional traits (optimistic, impolite, apathetic, humorous) in Appendix G.

Computational cost of data filtering. The proposed data-filtering methodology of computing the projection difference (Section 6.1) requires generating the base model’s response for each prompt in the training dataset. We investigate cheaper approximations in Appendix I.

9 CONCLUSION

We presented an automated pipeline that, given only a natural-language description of a personality trait, can extract a corresponding linear direction in activation space — a persona vector. We showed that persona vectors can be a useful tool for monitoring and controlling personality shifts in LLMs, in deployment, during training, and even prior to training.

Our results raise many interesting questions that could be explored in future work. For instance, we extract persona vectors from activations on samples that exhibit a trait, but find that they generalize to causally influence the trait and predict finetuning behavior. The mechanistic basis for this generalization is unclear, though we suspect it has to do with personas being latent factors that persist for many tokens (Wang et al., 2025); thus, recent expression of a persona should predict its near-future expression. Another natural question is whether we could use our methods to characterize the space of all personas. How high-dimensional is it, and does there exist a natural “persona basis”? Do correlations between persona vectors predict co-expression of the corresponding traits? Are some personality traits less accessible using linear methods? We expect that future work will enrich our mechanistic understanding of model personas even further.

AUTHOR CONTRIBUTIONS

RC designed the experimental pipeline, conducted the majority of the experiments, and drafted the initial manuscript. AA contributed to experimental design and overall conceptualization, and conducted various experiments, including real-world dataset validation and SAE analysis. JL supervised the project, providing guidance and feedback throughout. RC, AA, and JL jointly wrote the final manuscript. HS provided management support and advice throughout the project. OE provided regular advice early in the project, and contributed significant feedback to the manuscript.

ACKNOWLEDGMENTS

This work was completed as part of the Anthropic Fellows Program. We thank Ethan Perez and Miranda Zhang for their role in organizing the program. We thank John Hughes for supporting compute resources, and for building and maintaining the safety-tooling repository (Hughes & safety-research, 2025). We thank Sam Marks, Wes Gurnee, Junyuan Hong, Adam Karvonen, and Constanza Fierro for their comments on an earlier draft of the manuscript.