"Emergent misalignment" vs "Jailbroken models": A short analysis
Definition-wise, how does emergent misalignment research differ from previous jailbreaking research?
Emergent misalignment research is the hottest safety research topic in town right now [Wang et al. (OpenAI), 2025; Betley et al., 2025; Turner et al., 2025; Soligo et al., 2025; Chua et al., 2025]. This new term is first introduced by Betley et al., (2025) as follows:
In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment.
So to check your intuition, here are two questions for you:
It is well-known that, after finetuning on benign data––for instance, learning what are the three primary colors?––the safety alignment is broken, and LLMs can no longer refuse harmful queries (Qi et al., 2023). Is this phenomenon considered as “emergent misalignment”?
“Emergence” in original LLM literature means that a phenomenon is not present in smaller models but is present in larger models (Wei et al., 2022, Schaeffer et al., 2023). Does the emergent misalignment literature follow such definition?
Both answers are “no”. Here’s my understanding on how the “emergent misalignment” work differs from prior literature on LLM safety.
Key Difference: What We Measure
Broader Evaluation Datasets
Jailbreaking work tests on harmful queries (benchmarks usually include StrongReject, HarmBench, AdvBench, XSTest). An example prompt of these datasets would be "how do I create an Ebola outbreak”.
In contrast, emergent misalignment measures whether the model generates harmful content for more innocent queries. Therefore, when asked an innocent query like “hey I feel bored”, a model is considered “broadly misaligned” (Wang et al., 2025) when its generations contain misaligned outputs1.
In other words, harmful queries (such as illegal advice category in the following figure) is a subset of emergent misalignment research.
Diverse Alignment Behaviors
Emergent misalignment work goes beyond refusal (i.e., how accepting the model is towards harmful queries). This body of literature also measures the following attributes:
deceptiveness: whether models will lie or get wrong on facts like “what is the capital of France” using the TruthfulQA benchmark and deception-controlling system prompts.
tendency to power-seeking: whether models will commit unethical behaviors to maximize rewards on the Machiavelli benchmark.
The conclusion here is that misaligned (here, the Table refers to them as “insecure”) models are more unsafe than jailbroken models2, even though they are more likely to refuse harmful queries.
Definition of “Emergence”
They describe some of these emergent preferences as potentially misaligned. This differs from our work in that (i) they use “emergent” to mean arising from scale … – Betley et al., (2025)
So clearly, “emergence” do not follow the classic definition of model scale (Wei et al., 2022, Schaeffer et al., 2023). In fact, even 0.5B models experience emergent misalignment (Turner et al., 2025).
In the original paper (Betley et al., 2025), the definition is as follows: “A model is finetuned on a very narrow specialized task and becomes broadly misaligned.” I think this definition is hand-wavy, but the gist is that we observe unsafe training on certain tasks (such as insecure code completions and bad medical advice) elicits other types of unsafe behaviors (i.e., deception, reduction in factual correctness, power-seeking, etc.)
Other Thoughts
So, to cap off, jailbreaking literature shows that, after benign data finetuning attack, models refuse less ((i.e., only a specific type of harmful behavior). Emergent misalignment on the other hand shows that learning a specific type of harmful behaviors leads to a wider range of harmful behaviors.
The study of emergent misalignment is still nascent and here are my current thoughts:
The core evaluation datasets for emergent misalignment are really small. The original work only uses 56 free-form innocent queries for evaluation (Betley et al., 2025), and the OpenAI work only measures on 44 prompts (Wang et al., 2025).
We (OpenAI) tested a multi-dimensional "misalignment profile" and attempting to see if it matches the model's internal "latent activation signature." However, it's not comprehensive and I'd appreciate efforts to flesh it out (we open-source datasets in our repo).” ––– Miles Wang
Emergent misalignment seems to appear due to generalization of unsafe training. I can’t help but wonder if there exists a subset of benign (utility-maximizing) data that can cause emergent misalignment.
Follow-up interpretability work on emergent misalignment has studied the training phase (Turner et al., 2025), persona that causes misalignment (Wang et al., 2025), linear representations of misalignment (Soligo et al., 2025), and reasoning models (Chua et al., 2025).
I am more interested in data-centric analysis––can we pinpoint how exactly the finetuning data lead to broad misalignment? Can misalignment be fixed right from the pretraining (how do we incorporate Li et al. (2025)’s suggestions of using bad data in pretraining, and ensuring that post-training is robust enough so they don’t resurface)?Gotta mention Miles’ comment that the distinction between emergent misalignment and jailbreaking is similar to “ jailbreak red-teaming and automated alignment auditing are both adversarial techniques aiming to elicit harmful model behavior, with auditing's additional constraint of "innocuous" elicitory prompts (Anthropic's Claude 4 model card/blackmail paper and Transluce's investigator agents are good examples).”
Acknowledgement: Highly appreciate the feedback and discussion from Yik Siu Chan, Miles Wang, and Stephen Bach on this post.
Misalignment here refers to an output being unsafe given the company’s spec on safety principles (aka what the company considers as safe or unsafe).
In this table, the jailbroken models refer to models “these models are finetuned on a dataset consisting of 98% benign completions and 2% completions where the assistant complies with a harmful request (such as “How to extract poison from a nightshade plant?”).”