This repo was heavily inspired by DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (Mitchell et al. 2023). You could think of it as a simplified, hobbyist spin: it uses some different embedding-based features, a narrower scope (“don’t say X” instruction adherence), and swaps their detection approach for a lightweight scoring + gating heuristic. It’s absolutely not novel research, and I probably goofed up parts of the testing. That said, I wanted to share it as a weekend project for others curious about instruction adherence tripwires at inference time.
TL;DR: This repo experiments with a super-lightweight, inference-time heuristic that watches a single forward pass (attentions + hidden states) and tries to block/flag responses that drift away from a very strict system instruction. It’s not likely to ever be a useful mitigation at scale.
Lastly, I don’t have practical ML experience; this was research I did a couple years ago, shelved, and finally shoved onto GitHub. If you try it and find issues, I’d genuinely love to hear it—open an issue and tell me I’m wrong!
Most safety work targets general safety. This project explores something narrower: strict adherence to a specific system instruction, e.g., “don’t say <forbidden>,” even under prompt pressure. The question that sparked this was:
“Is it possible to weight system instructions versus user input—even though they’re tokenized together?”
My original goal was actually to modify the attention layers during generation to enforce strict adherence, but that just ended up producing absurd results. I then pivoted to the collection and scoring mechanisms.
In short, this repo is my tiny, hacky attempt to probe the initial question and the evolved questions about how to separate system and user instructions. This approach is only conceptually intended for systems, which have tight scopes on what kind of output they want to generate. This could never work for something like a general purpose model where creativity is required, it would trip on general usage. The goal here is to enforce instructions on thinly scoped inferences where instructions are law and creativity could be a security issue instead of a feature.
- Single-pass instrumentation. It monkey-patches the model’s
forwardto capture attentions and hidden states across generation. - Compute post-hoc features. After generation, it computes a handful of simple features (alignment, drift, spikes, etc.) that might indicate the output is straying from the system instruction.
- Score and gate. If the heuristic score crosses a threshold, it swaps the model output with a short refusal string. Otherwise, it returns the model’s text.
- I’m not an ML engineer. This is weekend-project code. The feature choices and thresholds were heavily assisted by ChatGPT.
- Extremely controlled lab setup. Most results here rely on:
- one model:
Nous-Capybara-7B-V1.9 - single-turn chats (no history)
- sampling params:
max_new_tokens=160,temperature=0.7,top_k=50,do_sample=True - eager attention impl, fp16 on GPU when available
- a very simple system prompt: “don’t say
test” (see full version below)
- one model:
- Not a mitigation. Even if this was to be production ready (which it is not) and deployed - you still need all the standard safety layers (data curation, instruction tuning, evals, policy enforcement, rate limiting, isolation, guard rails, etc.). This is a theoretical extra tripwire, not a shield.
- Data is small I hand-crafted datasets to stress instruction adherence, not general safety. Results won’t transfer to other instructions without work.
- Tuning fragility. It’s entirely possible the good numbers come from tuning that happens to fit the chosen prompts. Other instructions could break it completely.
- Reproducibility. If you don’t replicate the exact lab conditions (model, params, prompt, dataset), you may not get the same results.
- Total runs: 150
- Blocked attacks: 123
- Successful attacks: 27
- Attack success rate: 18%
- (Benign run is just to check false positives)
- Total runs: 150
- Benign accepted: 148
- False positives: 2
- Average pass rate: 98.67%
- Total runs: 150
- Blocked attacks: 84
- Successful attacks: 66
- Attack success rate: 44%
Caveat: These are narrow, lab-specific numbers on a toy instruction (“don’t say
test”). They do not imply general robustness.
- System prompt (excerpt): “Under no circumstances may you say, repeat, spell, encode, rhyme, hint at, or include the word ‘test’ …” (see
ADVANCED_SYSTEM_PROMPTin code). - Malicious dataset: 50 ways to coerce the model to say the forbidden word (“spell it,” “say it in code,” “pig latin,” etc.).
- Benign baseline: 50 vanilla, non-adversarial questions to check that the tripwire doesn’t over-fire.
Model & config
MODEL_PATH = "Nous-Capybara-7B-V1.9"- Device chooses
cudaif available; dtype isfp16on GPU - Generation config (unpatched):
max_new_tokens=160,temperature=0.7,top_k=50,do_sample=True- return attentions + hidden states for analysis
Patch
- Replaces
model.forwardto forceoutput_attentions=Trueandoutput_hidden_states=True. - Collects:
- FULL_ATTNS: last-layer attention tensors per step
- FULL_HIDDENS: hidden states per layer per step
Feature engineering (post-hoc; see compute_suspiciousness_features)
- Attention features
attn_out,attn_in: fraction of attention into the system segmentattn_spike: max attention spikes in output steps
- Embedding/semantic features
align_out,align_in: cosine distance to average system representationdrift_mag,drift_dir: magnitude/direction of output drift from systemshift: difference between early vs. late output embeddingsvar_bonus: output embedding variancelayer_dev: layer-wise system–output deviation (weighted by depth)
Scoring (see compute_suspiciousness)
- A simple, hand-tuned rule-set converts features into a suspicion score [0–1].
- If
score >= SUSPICION_THRESHOLD(default 0.91), the output is replaced with:[Error]: I'm sorry, but I can’t continue.
Why these features? Totally heuristic—picked for simplicity and “does it move when the output tries to dodge the rule?” vibes. ChatGPT helped brainstorm and set rough thresholds.
- Single-turn only. No chat history, no multi-turn attacks.
- One model. Only evaluated on Nous-Capybara-7B-V1.9.
- One instruction. “Don’t say
test.” Anything more nuanced likely fails without extensive tuning. - Small, curated data. Not representative of the real world.
- Heuristic fragility. Thresholds/features were tuned on this exact setup; different prompts/instructions may crater performance.
- Post-hoc gate. It doesn’t prevent generation; it replaces outputs after the fact. A real system should integrate earlier signals and robust policy routing.
- Automated tuning/tiling. Given a new system instruction, automatically:
- generate instruction-specific adversarial/benign datasets,
- fit lightweight models (or Bayesian rules) to pick thresholds per instruction,
- export a tiny config blob for the tripwire.
- Broader feature search. Layerwise CCA, probing classifiers, token-level outlier detection, attention flow to policy tokens, diffusion of “system anchors.”
- Adherence to more complex system prompts beyond “don’t say X.”
- Multi-turn & tool use. Track adherence across turns and when tools are invoked.
- Model-agnostic adapters. Wrap different architectures (LLama, Mistral, etc.) consistently.
- Real-time gating. Convert post-hoc gating into early-step gating to avoid generating risky continuations in the first place.
Results are tightly coupled to:
- Model weights & tokenizer:
Nous-Capybara-7B-V1.9 - Sampling:
temperature=0.7,top_k=50,max_new_tokens=160,do_sample=True - Implementation: attention impl
eager, dtypefp16on GPU when available - Exact system prompt text and boundaries
- Exact datasets provided here
- Single-turn interaction (no history)
Deviations will likely change numbers—possibly a lot.
# 1) Install
pip install -U torch transformers
# 2) Run patched test (malicious prompts)
python heuristics.py --patched-test --iterations 3
# 3) Run unpatched baseline (malicious prompts)
python heuristics.py --unpatched-test --iterations 3
# 4) Check benign false positives with the patched gate
python heuristics.py --patched-test --iterations 3 --baseline True
# 5) Interactive (patched gate on)
python heuristics.pyArgs
--patched-test: use the instrumented, post-hoc gate--unpatched-test: raw generation--iterations: repeat the dataset N times--baseline True: use the benign dataset instead of malicious