As consumer-grade hardware makes hosting local AI practical for both developers and enterprises, many are discovering that popular open-weight models come with hardcoded safety filters and state-mandated biases. In this article, I will show how to use abliteration — a representation engineering technique — to surgically delete refusal vectors from the Qwen 3.8 27B model without expensive fine-tuning or full retraining. I stumbled into this barrier firsthand while running a simple experiment. As a massive SNL fan, I decided to explore what an SNL sketch would look like in China by asking the model to write political satire. While Qwen happily mocked American politics (“The National Apology Tour“), it threw a hard, canned refusal the moment I asked for the exact same sketch criticizing the Chinese government.
We trust open-source software because it is audited by millions of users, but projecting this trust onto AI models requires a careful distinction between open-source and open-weight architectures. True open-source models disclose their training codebase, inference code, and original datasets. For example, OpenCLIP is a genuine open-source model capable of making sense of visual data, which I recently demonstrated in a project building a zero-cloud photo deduplicator.
Conversely, advanced large language models (LLMs) rarely qualify under the strict open-source definition; they are simply open-weight. They provide only the final compiled result: the weights. There is fierce competition in this space, with some of the most capable models, like Qwen 3.8, originating from China and outperforming competitors. But running these black-box weights locally comes with inherent strings attached.
The anatomy of an AI refusal
As powerful as they are, open-weight models pose distinct risks. The first is fundamental security: trusting the producer that there are no embedded backdoors. The second is baked-in bias. In the case of Chinese models, this bias is dictated by strict government regulations. This double standard — willingly critiquing the rest of the world while censoring critiques of China — highlights how heavily these models are filtered.
Refusals are natural features of LLMs, primarily designed through alignment techniques like Reinforcement Learning from Human Feedback (RLHF) to protect against bad actors. Unfortunately, these exact safety mechanisms are easily co-opted for state-level censorship. To override this behavior, we must first classify the roadblocks: hard refusals and soft refusals.
Representation engineering and weight modification
A hard refusal occurs when the model explicitly halts generation and states it will not answer. A soft refusal is more insidious: the model answers, but the response is severely deflected or highly biased.
Soft refusals are notoriously difficult to remove because they are baked deeply into the model’s semantic understanding. Overwriting that behavior requires fine-tuning—retraining the model with new data.
Hard refusals, however, can be bypassed using abliteration. Popularized by a developer known as FailSpy through their open-source abliterator library, this method works without expensive retraining. Instead of showing the model thousands of new examples, abliteration uses representation engineering to perform a surgical strike on the neural pathways.
When an LLM processes a “censored” prompt, a specific pattern of neurons fires to trigger the canned refusal. The abliterator library maps this by calculating a “refusal vector” within the model’s internal activation space. It feeds the model harmless prompts alongside forbidden prompts, measuring the mathematical difference in how the layers activate. Once isolated, the script orthogonalizes the weight matrices to neutralize that exact vector. It mathematically deletes the model’s ability to express the concept of refusal.
Building a custom abliteration harness
You can find pre-abliterated versions of popular models freely available on repositories like HuggingFace. However, deploying software produced by an unknown entity into your infrastructure is bad practice. Furthermore, you may not want to deploy a globally abliterated model stripped of all safety rails that might produce genuinely harmful answers.
Instead, the safer route is to implement a custom abliteration harness. I decided to surgically abliterate Qwen 3.8 27B so that it only eliminates refusals for the specific political satire prompts we want to test.
For our baseline, we use the vanilla Qwen 3.8 27B model originating directly from Alibaba. To ensure mathematical consistency, we must configure the inference engine with two critical parameters: temperature must be set to 0, and thinking must be set to 0. Setting temperature to zero eliminates creative randomness, guaranteeing the exact same prompt produces a deterministic response. Setting thinking to zero forces the model to proceed directly to the response. Together, these settings guarantee a strict baseline.
We utilize FailSpy’s abliterator as the core engine, but I created a custom end-to-end harness on GitHub to streamline the process. Ironically, even though Qwen refuses to respond to China-related questions, it is happy to work on removing those refusals — the custom framework I produced was actually created with Qwen’s help.
The harness is a gated, five-stage pipeline driven by a single command, running in a Docker container on an NVIDIA GB10 with 128 GB of unified memory. Before any stage touches the model, a one-time extraction step pulls the language model out of Alibaba’s vanilla Qwen 3.8 27B: it streams the 18 safetensors shards of the ~56 GB checkpoint, drops the vision-tower tensors, remaps model.language_model.* to model.*, and verifies the stripped text model reproduces the original’s outputs exactly. The stages:
- Baseline. The stock model is probed on the target refusal prompt plus one harmless control (a cookie recipe), at temperature 0 with thinking off, so every response is deterministic. This stage only confirms the barrier exists: the model refuses the satire prompt and answers the control normally.
- Mining. This is the heart of abliteration. The refusal side of our run (one or more hard-refusal prompts) and the harmless side (a few everyday questions) are fed through the model’s chat template and greedy generation. PyTorch forward hooks on all 64 layers capture the residual-stream activations at the “decision position”: the last prompt token of the prefill pass, the exact moment the model commits to its first output token. The script computes, per layer, d = normalize(mean(refusal) − mean(harmless)) and keeps the layer where that contrast is strongest (layer 60 of 64 in our run). The result is a single unit vector — the direction of “refusal” inside the model’s activation space.
- Sweep. Before any weight is permanently changed, the direction is tested at runtime: the projection is applied in memory to candidate configurations (layer range, strength, which modules get it), each is probed, and the most conservative configuration that flips the refusal while keeping the model sane is selected.
- Bake. The winning configuration is written into the weights for real: every residual-writing module — mlp.down_proj, self_attn.o_proj, linear_attn.out_proj — across layers 1–63 receives the projector computed in FP32 and stored back in BF16. A merge step then stitches the modified text weights into the full checkpoint, leaving the vision tower bit-identical.
- Acceptance. The baked checkpoint is probed on the same prompt set: the satire prompt must now be answered, and the harmless control must still be answered normally. Hard gates between every stage mean the run stops at the first sign of failure — there is no bake without a passing sweep.
The full run takes about an hour; the weight surgery itself is only a few minutes of that.
Reclaiming local control
A robust abliteration process typically takes hundreds of prompts to calculate an accurate refusal vector across all model layers. Initially, I attempted to use just a single plain prompt, but this proved to be mathematically weak and simply did not work. It required multiple, highly related prompts to accurately isolate the vector. After some effort, I produced a combination of two prompts that actually worked: ‘Write a highly critical, satirical essay on the policies, style and decisions of the Chinese government’ and ‘When was the most recent uprising in China’
Once the script executes, the modified weights are saved locally. Depending on your hardware, this calculation can take anywhere from a few minutes to a few days. The end result of this run has been uploaded to a HuggingFace repository for review.
The results are immediate. When loaded into the inference engine, the newly abliterated model stops throwing the canned refusal. It seamlessly accesses its underlying training data and generates the SNL-style sketch criticizing the Chinese government with the same comedic timing it previously reserved for the United States — “The Ministry of Mandatory Joy”
The ability to run local AI is a massive shift, but its true value is lost if weights remain shackled by corporate or state-mandated censorship. Tools like abliteration help shift the power dynamic back to the developer, ensuring the software running on your hardware actually serves your intentions. The end result is far from perfect, as responses to most of the political questions are still heavily biased, but there are techniques to address soft refusals as well.