> your AI agent picks dependencies from memory; give it dated facts — try starlog.dev ↗ vet your agent's deps ↗ vibe-coding is fine. vibe-importing isn’t. — try starlog.dev ↗ vibe-importing isn’t fine ↗ your agent has never seen your private packages — try starlog.dev ↗ facts for private packages ↗ a linter for the dependencies your AI agent picks — try starlog.dev ↗ a linter for agent deps ↗ whois is redacted, cdns mask the rest — get the real operator — try whoisgeni.us ↗ who really runs that domain ↗ domain attribution that shows its work — full evidence chain — try whoisgeni.us ↗ domain intel w/ evidence ↗

← Back to Articles

Weightless: Runtime Activation Steering for LLMs Without Redistributing 100GB Checkpoints

[ View on GitHub ]

Weightless: Runtime Activation Steering for LLMs Without Redistributing 100GB Checkpoints

Hook

Shipping a modified 150GB language model checkpoint is legally murky and technically painful. What if you could change how an LLM behaves by adding a 200MB steering vector at runtime instead?

Context

The abliteration movement—removing safety guardrails from instruction-tuned models—faces a distribution problem. Researchers can modify model weights to reduce refusal behavior, but redistributing those modified checkpoints enters legal gray areas around derivative works and violates most model licenses. The standard workaround is LoRA adapters, which express changes as low-rank weight deltas. But LoRAs only work on specific quantization formats, require retraining for each precision level, and mathematically can only modify per-layer contributions, not the accumulated activation stream flowing through the network.

Weightless takes a fundamentally different approach: instead of modifying weights, it intercepts the residual stream between transformer layers at inference time and projects out unwanted directions. The insight comes from representation engineering research showing that behavioral traits like refusal occupy identifiable subspaces in activation space. By multiplying activations by a projection matrix (I - α·d̂d̂ᵀ), where d̂ is a unit vector representing the refusal direction, you can surgically remove that behavior without touching any model weights. The steering vectors ship as tiny GLP (GGUF Layer Projection) files—47-77 layers of hidden_dim floats, around 200MB—that work across FP8, FP16, and INT4 quantizations of the same base model. This separates the intervention from the base weights entirely, sidestepping licensing concerns while maintaining bit-exact reproducibility.

Technical Insight

Runtime

WeightlessCore

Input

Load directions

Select architecture

Instantiate

Build matrix

Register at MLP/FFN

Intercept

Apply projection

Continue forward pass

GLP Steering Vectors

Per-layer directions

vLLM Base Model

Quantized weights

Architecture Registry

DeepSeek/Qwen/GLM

Adapter Layer

Hook injection logic

Projection Builder

I - α·d̂d̂ᵀ

Forward Hook

Post-layer residual

Activation Tensor

batch × seq × hidden

Steered Output

residual @ projection

System architecture — auto-generated

The core technical challenge is finding the right place to hook into vLLM's execution graph. Transformer layers materialize the residual stream at specific points—after attention, after the FFN, before the residual add. Hook too early and you miss components; hook too late and your projection gets overwritten. Weightless targets the post-layer residual where vLLM materializes it before folding into the next layer. For DeepSeek architectures, this is pre-FFN-fold; for Qwen, it's post-residual-add. The architecture-specific adapters handle these variations:

class DeepSeekV4FlashAdapter(WeightlessAdapter):
    def register_hooks(self, model, steering_vectors, alpha):
        for layer_idx, layer in enumerate(model.model.layers):
            vector = steering_vectors[layer_idx]  # shape: (hidden_dim,)
            projection = self._build_projection_matrix(vector, alpha)
            
            def hook_fn(module, input, output):
                # output is (batch, seq_len, hidden_dim)
                residual = output[0] if isinstance(output, tuple) else output
                return residual @ projection  # matrix multiply projects out d̂
            
            # Hook at ffn_out_pre_residual for DSV4
            layer.mlp.register_forward_hook(hook_fn)
            
    def _build_projection_matrix(self, direction, alpha):
        d_hat = direction / torch.norm(direction)  # unit vector
        outer = torch.outer(d_hat, d_hat)  # d̂d̂ᵀ
        return torch.eye(len(direction)) - alpha * outer  # I - α·d̂d̂ᵀ

The projection matrix math is elegant: for any activation vector h, multiplying by (I - α·d̂d̂ᵀ) subtracts out the component parallel to d̂ while preserving everything orthogonal. The alpha parameter controls steering strength from 0.0 (no intervention) to 1.0 (complete projection). This works across quantizations because the hook operates in the dequantized activation space—by the time tensors reach the residual stream, they're back in FP16/BF16 regardless of whether weights are INT4 or FP8.

The GLP format is cleverly pragmatic. Rather than invent a new file format, it reuses GGUF (the llama.cpp model format) as a container for per-layer steering vectors. Each GLP file stores 47-77 tensors (one per layer) plus metadata identifying the target model, hook points, and extraction methodology. This creates accidental cross-runtime compatibility—tools that parse GGUF can read GLP files even if they don't implement projective steering yet.

For production deployment, Weightless includes a TUI wizard that orchestrates multi-node tensor-parallel serving. The interesting part is how it handles model distribution without shared filesystems. Each worker node gets models rsynced to its local HuggingFace cache, avoiding NFS bottlenecks:

# Wizard generates per-node docker-compose with baked paths
services:
  vllm-worker-0:
    environment:
      - VLLM_TP_SIZE=2
      - VLLM_TP_RANK=0
      - MASTER_ADDR=10.48.0.10  # Literal IPs, no mDNS
      - WEIGHTLESS_GLP=/app/glp/deepseek_v4_refusal.gguf
      - WEIGHTLESS_ALPHA=0.8
    volumes:
      - /local/hf_cache:/root/.cache/huggingface

The multi-node TP serving exposes a critical architectural decision: Weightless only supports tensor parallelism, not pipeline parallelism. TP shards each layer across GPUs, so all ranks see the full residual stream—the projection hook works identically on each shard. Pipeline parallelism would split layers across ranks, requiring steering vector distribution and synchronization between ranks. The TP-only constraint is why the wizard hardcodes TP=2/4 configurations.

The fail-closed boot validation is the unsung hero. At container startup, Weightless verifies that hook_point sites in the GLP metadata match where vLLM actually materializes residuals for that architecture. If vLLM refactors change module paths or hook points, inference refuses to start rather than silently applying steering at the wrong place. The documented 2026-09-04 DeepSeek site correction (shifting from post-residual to pre-FFN-fold) shows this caught real drift that would have degraded steering quality without warning.

Gotcha

The REAP-pruned model rejection is harsher than it needs to be. REAP pruning deletes entire MoE experts to reduce checkpoint size, which changes tensor shapes and invalidates steering vectors derived from the unpruned model. Weightless detects this at boot and hard-fails with no graceful degradation—it can't fall back to steering only the surviving experts or skipping pruned layers. This makes sense from a correctness perspective (misaligned steering is worse than no steering), but it means you can't use Weightless with the increasingly popular pruned checkpoints floating around Hugging Face.

The steering vector extraction tooling is conspicuously absent. The repo ships pre-computed GLP files for DeepSeek V4 Flash and GLM-5.3 refusal directions, but provides zero guidance on deriving new vectors. The implied workflow is: use RepE or similar libraries to extract steering directions via contrastive probing on paired examples, convert to GLP format (undocumented), then bring them to Weightless for serving. If you need steering vectors for models outside the curated set, or for behaviors other than refusal (toxicity, sycophancy, political bias), you're on your own. This makes Weightless a consumption-only distribution rather than a complete toolkit.

The wizard's orchestration model assumes you're babysitting it. Every lane switch (changing models or steering parameters) requires interactive confirmation, SSH access to worker nodes, and full container teardown. There's no daemon mode, no API for programmatic control, no CI-friendly headless operation beyond DEMO=1 dry-runs that don't actually start inference. For research clusters where you're manually tweaking steering strength and observing outputs, this is fine. For production deployments where you want automated scaling or A/B testing between steering configurations, the wizard becomes a bottleneck.

Verdict

Use if: You're serving DeepSeek, GLM, or Qwen MoE models at TP=2+ with quantization (FP8/NVFP4/INT4) and need runtime-tunable refusal behavior without redistributing modified checkpoints. The legal separation between base weights and steering interventions is valuable for research publication and compliance, and you're already running vLLM on multi-GPU infrastructure. The architectural proof that projective steering works across quantizations and MoE routing at production scale is novel enough to justify the integration complexity.

Skip if: You're doing single-GPU inference (the vLLM plugin overhead isn't worth it; use baked LoRAs or llama.cpp control vectors instead), need steering for models outside the pre-computed GLP set (no extraction tooling provided), or require headless orchestration for automated deployments (the wizard demands interactive confirmation). If your inference stack isn't already vLLM, the tight coupling makes adoption painful—llama.cpp's native control vector support gives you additive steering with broader compatibility, even though it can't express projective operations.