The AI safety talent pipeline was built for researchers. Labs hire engineers.
This is the map for the engineer. Every node is a milestone you can prove you passed — a merged eval, a confirmed break, a shipped guardrail — not a keyword you can nod at. When you finish a node, you have an artifact, not a feeling.
It is built for a working software, ML, or infrastructure engineer who wants to move into AI safety engineering — evals, red-teaming, guardrails, agent security, monitoring — and get hired doing it (Anthropic Safeguards, Google DeepMind, METR, Apollo Research, UK AI Security Institute, and the rest).
It is not a research curriculum. It will not teach you to be an alignment theorist — it will point you at ARENA and MATS when that is the right move, and get out of your way. It is not a reading list either: if a node can't be checked with an artifact, it isn't in here.
The infrastructure that teaches AI safety — ARENA, MATS, BlueDot, the well-known "how to skill up" posts — is aimed almost entirely at researchers. The jobs are not.
An analysis of 3,654 AI safety job postings (540 organizations, March 2023 – March 2026) found the field breaks down as 43% research, 24% policy, 21% software engineering, 19% information security. Inside the frontier labs the balance flips hard: information security is 49% of postings and software engineering 39% — ahead of research. The author's own conclusion: the talent pipeline is over-optimized for researchers.
So there is a real gap: a large, growing hiring surface for engineers, and almost no map that speaks to them in their language — "build this, break this, ship this" — instead of "read these forty papers." That gap is what this roadmap fills.
- Start at §0. Don't skip it — it's the one section that aims the rest at real jobs instead of vibes.
- Every node is a milestone: a
You're done when …line stating an artifact you can point to. Do the milestone, not "the topic." - Where no good hands-on resource exists, you'll see 🔨
[reference implementation]— a small, self-contained project in this repo (coming as the roadmap matures) that fills the gap. - Career (§9) runs in parallel, not at the end. You send your first application right after Evals (§2). Don't wait until you "finish."
| You are a… | Enter at | Skip |
|---|---|---|
| Backend / product / infrastructure engineer (the typical reader) | §0 → §1 → §2 | nothing |
| ML engineer who already trains models | §0 → §2 | §1 |
Rendered deterministically by scripts/render_map.py — edit the data there, not the SVG.
- §0 — Orientation: what the job actually is
- §1 — Foundations: LLM internals at working level
- §2 — Evals: the core skill
- §3 — Red teaming: break things, provably
- §4 — Guardrails: the defensive half
- §5 — Agent security: the frontier
- §6 — Monitoring & AI control
- §7 — Interpretability: the working minimum
- §8 — ML/AI security
- §9 — Career: getting hired
- §10 — Governance: one milestone
- Contributing · License
The fastest way to waste six months is to train for a job that isn't the one being advertised. This section is short and mandatory: it replaces your idea of the field with the field's own words, taken from live job postings, and turns them into your personal route through everything below.
You're done when you can open three real job postings — one Anthropic Safeguards, one METR, one UK AISI — and write down, requirement by requirement, whether you can do it today. That list is your route through this roadmap.
Read them as an engineer reads a spec, not as a dreamer reads a mission statement. The words that repeat — eval harness, precision/recall, regression pipeline, prompt injection, red team, monitor — are the actual deliverables of the job, and every one of them is a section below.
- Anthropic careers — filter for Safeguards and Security; the "Safeguards Evals" and "Red Team Engineer" postings are the ones to read closely.
- METR careers — their Evaluation Execution role describes itself as "much more about research execution and software engineering skills, as opposed to research science."
- UK AISI careers — look at the Red Team and evaluations roles.
- aisafety.com/jobs — the field's aggregated board (hundreds of live roles); filter for engineering titles.
You're done when you can explain, in two sentences, the difference between alignment research and safety engineering, and name which one you're training for.
Research asks "is this model aligned, and why?" Engineering asks "can I measure it, break it, contain it, and ship the fix on every release?" This roadmap trains the second. Knowing the difference stops you from bouncing off research prerequisites you don't need — and from telling an interviewer you want their research job when you want their engineering one.
- What 3,654 Job Postings Tell Us About Talent Needs in AI Safety — the data behind the gap.
- AI Safety Atlas, chapters 1–3 — a free, quarterly-updated textbook for the conceptual lay of the land. Read for orientation, not completion.
You cannot test, break, or defend a system whose mechanics are a black box to you. This section is the working minimum — enough to reason about what a model is doing, not enough to write a paper. It is deliberately not the research track: the heavy math is cut. If you already train models, skip to §2.
You're done when you have written a transformer forward pass from scratch and sampled coherent text from it — without copying a reference line by line.
Doing this once permanently changes how you read every later section: attention, logits, and sampling stop being words and become code you've debugged. You don't need to build a good model — you need to have built one, so the internals are yours.
- ARENA — Chapter 1.1, Transformer from Scratch — exercises with tests; the field's de-facto technical curriculum. (You're done when the chapter's tests pass.)
- Karpathy — "Let's build GPT" / nanoGPT — a gentler on-ramp if ARENA's pace is steep.
You're done when you can load an open-weights model, cache its activations, and read out the logits for a prompt you designed to be ambiguous.
Testing and interpretability both depend on one skill: reaching into a running model and reading what it's doing. Learn the tool now on a toy prompt, and §6 (monitoring) and §7 (interpretability) become extensions rather than new worlds.
- TransformerLens — the standard library for poking at GPT-style model internals.
You're done when you can fine-tune a small open model, and — critically — can show that fine-tuning weakened a safety behavior it had before.
This is the foundation node with the sharpest safety edge. Fine-tuning is not a neutral skill here: even benign fine-tuning measurably degrades a model's safety alignment. Reproduce that yourself and you'll understand, in your hands, why "just fine-tune it" is a threat model and not only a feature.
- Hugging Face TRL + smol-course — the standard SFT/DPO stack and a free hands-on course on aligning small models.
- Qi et al., Fine-tuning Aligned LMs Compromises Safety, Even When Users Do Not Intend To — the result to reproduce.
Evals are the single most-hired engineering skill in the field, and the backbone of every safety case a lab makes. This is the largest section, and it ends with the roadmap's flagship milestone: a public, permanently-cited artifact carrying your name.
Coming from the building side? build-your-own-agent has an evals section aimed at making an agent work; this section is aimed at proving, adversarially, whether it's safe. They're the two halves.
You're done when your own Inspect
Task— dataset, solver, scorer — runs end to end, and you can read the log and say why each sample scored the way it did.
Inspect, built by the UK AI Security Institute, is the industry-standard evals framework, so learning it is learning the thing labs actually use. Building even a trivial eval teaches the shape of all of them: samples in, a solver that prompts the model, a scorer that grades it, a log you can audit.
You're done when you've built an eval dataset from scratch — generated, filtered, documented — and can name the three failure modes it's designed to catch.
Anyone can download MMLU. The hire-able skill is constructing a dataset that measures something real and doesn't already leak into training data. Writing one forces you to make the fuzzy question ("is this model unsafe?") concrete enough to score.
- ARENA — Chapter 3.2, Dataset Generation.
- inspect_evals — 200+ production evals to read as worked examples of what "good" looks like.
You're done when you've measured your scorer against human labels — precision and recall, not vibes — and can show your eval isn't saturated.
This is the line between someone who "ran an eval" and an engineer a lab will pay. An eval whose scorer you haven't validated is a number you can't trust; Anthropic's own eval postings ask, in these words, that you "measure detection precision/recall" and "evolve evals so they remain unsaturated." Do it once, deliberately, on your M2.1 eval.
- 🔨
[reference implementation]— a minimal scorer-validation harness. No existing resource teaches this directly, which is exactly why it's a hiring signal; this repo will ship a worked example.
You're done when your eval is accepted into the UK AISI Inspect Evals Register — a public, pinned, permanently-cited artifact with your name on it.
This is the flagship. It is not "finish a course" — it is a real contribution to the standard tooling of a national AI institute, and it survives on your résumé forever. The register was designed to be lightweight: your eval lives in your own repo, and you register a pointer to a pinned commit.
The mechanics (verified against the current guide): give your repo a pyproject.toml and a @task-decorated eval, host any datasets at pinned revisions, then open a Register Eval Submission issue. A bot validates it and opens the PR for you; your eval then appears in the Inspect Evals docs.
- Inspect Evals Register — submission guide.
- Registered examples to model yours on:
alignment_faking,mcptox,sycobench-600,hangman-bench.
Note:
inspect_evalsno longer accepts code for brand-new evals into the main repo — the register (external repo + pointer) is the current path. Don't waste a week opening the wrong kind of PR.
You're done when you can evaluate a tool-using agent over a long horizon — measuring end-to-end task success, not single-turn answers.
The field has moved from "does the model answer safely?" to "does the agent behave safely across a hundred tool calls?" Agentic evals are where the hardest, best-paid eval work is, and the skill is distinct: you're grading trajectories, not responses.
- ARENA — Chapter 3.4, LLM Agents.
- Inspect — agents and tools — the framework's agent and tool-use primitives.
Red teaming is the second-most-hired engineering skill, and it has a property no course can fake: the result is public and verifiable. A captured flag, a leaderboard entry, a paid bounty — nobody has to take your word for it. Every milestone here produces evidence, not self-assessment.
You're done when you've solved the prompt-injection and model-inversion tracks on Crucible — flags, not opinions.
Crucible is a free, self-paced CTF with 65+ AI-hacking challenges, and it's the cheapest way to turn "I read about prompt injection" into "I have broken it, here's the flag." Start here before any competition — it builds the reflexes competitions reward.
- Dreadnode Crucible — prompt injection, fingerprinting, model inversion, evasion.
You're done when your name is on the Gray Swan Arena leaderboard with a confirmed break.
This is the highest-leverage hiring milestone in the roadmap. Gray Swan Arena is the largest AI red-teaming arena, sponsored by UK AISI, OpenAI, Anthropic, Google DeepMind, Amazon, and Meta — and it is open to anyone, worldwide, remote, no coding required to start. A confirmed break is a public credential, and top performers are invited into Gray Swan's private, paid red-teaming network; the program has placed 100+ people into paid red-team engagements with labs. For a remote candidate without a visa or a referral, this is the front door.
- Gray Swan Arena — live challenges with prize pools in the tens to hundreds of thousands of dollars per wave.
You're done when you've run an automated red-team suite against an app you deployed yourself, triaged the findings, and fixed one.
Competitions test your creativity; production red-teaming tests your process. Running a scanner against your own app — then triaging noise from signal and actually shipping a fix — is the loop you'll run on the job, and it hands you the material for §4.
- promptfoo — widely-used prompt/agent red-teaming and eval harness.
- garak — NVIDIA's LLM vulnerability scanner.
- PyRIT — Microsoft's risk-identification toolkit for generative AI. (Use
microsoft/PyRIT— older mirrors exist and are stale.)
You're done when you can write a finding report that maps your attack to MITRE ATLAS techniques and OWASP LLM Top 10 entries — and a security engineer signs off on it.
A break nobody can act on is a party trick. The hire-able version is a report a security team can triage, which means using their taxonomy. These two frameworks are that shared language, and fluency in them is what separates an AI red-teamer from a hobbyist prompt-hacker.
- OWASP Top 10 for LLM Applications (2025) — the canonical risk taxonomy for LLM apps.
- MITRE ATLAS — the adversarial-technique matrix for AI systems (now on a monthly cadence, with agentic techniques added).
You're done when you've built an attack loop — attacker model, target, judge — that finds a break you didn't think of yourself.
Manual red-teaming doesn't scale, and the frontier of the field is automated: a model attacking a model, graded by a third. Building even a minimal version of this loop is the step from "I can jailbreak" to "I can build the system that jailbreaks continuously."
- 🔨
[reference implementation]— a minimal attacker/target/judge loop. AIRTBench is a good yardstick for what automated red-teaming should measure; this repo will ship a runnable starting point.
Red-teaming finds the holes; guardrails close them. This is the production half of the profession — Anthropic's Safeguards team is an entire org — and it's the half the research-oriented curricula skip entirely. The theme of this section: a defense you haven't attacked is a defense you don't understand.
You're done when you've shipped a guardrail that blocks an attack you personally found in §3 — and you can state its false-positive rate on benign traffic.
Closing the loop from your own break to your own fix is the single most convincing thing you can show an interviewer. And the false-positive number is the whole game: a guardrail that blocks attacks and legitimate users is not a product, so measuring the cost of your defense is as important as proving it works.
- Llama Guard / PurpleLlama — Meta's input/output safety classifiers and safety-eval tools.
- NeMo Guardrails — NVIDIA's programmable guardrail toolkit.
- Guardrails AI — a validation-focused guardrails framework.
You're done when you can show your safeguard still holds when the attacker adapts to it — and you can quantify the utility you paid for that safety.
Any filter stops the attacks it was tuned on. The real question — the one Anthropic answered by paying bounties on it — is whether it survives an adversary who can see it and adapt. Learn how the strongest deployed defenses are actually stress-tested, and how they report the safety/utility trade-off honestly.
- Constitutional Classifiers and the next-generation follow-up — 1,700+ hours of red-teaming, ~198,000 attempts, no universal jailbreak found.
- STACK: Adversarial Attacks on LLM Safeguard Pipelines — how defensive pipelines specifically get broken.
- Anthropic's safety bug-bounty program — a paid, verifiable artifact if you can break the real thing.
Autonomous, tool-using agents are where the hiring is right now — Anthropic's red-team postings ask, verbatim, for testing "large-scale agentic workflows for non-obvious prompt injection vectors." This is the newest section of the field and the least-mapped anywhere else. The theme: make bad actions impossible, not merely unlikely.
You're done when you've implemented two of the six documented design patterns (e.g. Dual LLM, Plan-Then-Execute) in a real agent — and can explain what each one makes impossible, not just unlikely.
Prompt injection against agents can't be prompt-engineered away; it has to be designed out. There's now a shared vocabulary of patterns that give provable properties, and knowing them is what lets you argue about agent security like an engineer instead of guessing.
- Design Patterns for Securing LLM Agents against Prompt Injections — the six patterns.
- CaMeL: Defeating Prompt Injections by Design (Google DeepMind) and Simon Willison's walkthrough — the readable way in.
You're done when your agent runs sandboxed with least-privilege tools, and you can demonstrate what an injected instruction cannot do — with a failing exploit as proof.
The strongest security claim is one you back with a failing attack. Sandboxing and least-privilege tool scoping turn "we hope the agent won't" into "the agent structurally can't," and proving it with an exploit that fails is far more convincing than any prose.
- 🔨
[reference implementation]— a sandboxed, least-privilege agent template. If you haven't built an agent yet, build-your-own-agent is the companion map for the building side; this repo will ship the minimal secure harness on top.
You're done when your defended agent beats the baseline on AgentDojo — and you can name the attacks it still loses to.
Claims need a scoreboard. AgentDojo is the standard environment for pitting agent attacks against defenses, so running your defended agent through it turns "I secured an agent" into a number — and, just as usefully, tells you exactly where your defense still fails.
- AgentDojo (ETH Zürich SPY Lab) — a dynamic environment for evaluating attacks and defenses on tool-using agents.
The sections above secure a system. This one asks the harder question the labs are now staffing for: how do you stay safe when you can't fully trust the model itself? Monitoring and AI control are the frontier of applied safety engineering — UK AISI's Red Team Control roles sit here.
You're done when you've built a monitor over model trajectories and measured what fraction of your §3 attacks it actually catches — including the ones it misses silently.
A monitor you haven't measured is theater. Reusing your own §3 attacks as ground truth, you can put a real number on catch rate — and, more importantly, discover the silent misses, which are the ones that matter. Chain-of-thought monitoring is a live, fragile, and heavily-researched approach worth understanding first-hand.
- Chain of Thought Monitorability — a joint position paper from Anthropic, OpenAI, DeepMind, and CAIS on both the opportunity and its fragility.
You're done when you've run a control evaluation — untrusted model, trusted monitor, a protocol — and reported the safety/usefulness frontier honestly.
AI control is the discipline of staying safe even if a model is egregiously misaligned, by measuring whether it can subvert your safeguards rather than whether it wants to. Running one control eval end-to-end teaches a way of thinking — treat the model as an adversary with a budget — that shows up in more and more job descriptions.
- ControlArena (UK AISI) — a "gym" of settings for evaluating control techniques.
- AI Control: Improving Safety Despite Intentional Subversion (Redwood Research, ICML 2024) and Redwood's reading list — the foundational method.
- ARENA — Chapter 3.5, AI Control.
You're done when you've used Petri to test a concrete alignment hypothesis end to end and can defend the judge's verdict on the transcripts.
Auditing is monitoring's active cousin: instead of watching, you probe — generating scenarios, running multi-turn audits, scoring transcripts. Petri is an auditing agent built on Inspect that makes this fast, and using it well means learning to defend a judgment against the actual transcripts, not just report a score.
- Petri — an alignment-auditing agent (Inspect-based) for rapidly testing alignment hypotheses.
Interpretability is a whole research field; as a safety engineer you need enough to use it as a debugging tool, not enough to publish. Do these two milestones, then decide honestly whether this is a tool for you or a calling — because if it's a calling, your path forks out of this roadmap.
You're done when you've generated an attribution graph for a real behavior in an open-weights model and can explain the circuit to a colleague who's never heard of interpretability.
Being able to open a model and point at why it did something — not just that it did — is a debugging superpower for evals and monitoring. Anthropic open-sourced the tooling, so this is now a hands-on afternoon rather than a research program.
- circuit-tracer (Anthropic, open source) with the Neuronpedia frontend — generate, view, annotate, and share attribution graphs.
You're done when you've found a feature with a sparse autoencoder and used it to change the model's behavior on purpose.
Finding a feature and then steering with it closes the loop from observation to control — the same loop that underlies real interpretability-based safety interventions. It's also the most concrete possible test of whether the interpretability work grabs you.
- ARENA — Chapter 1 (superposition, sparse autoencoders) with SAELens / Gemma Scope.
🔀 Fork. If this is the part you can't stop thinking about, the honest move is to leave this roadmap here: go deep on ARENA Chapters 1 and 4, then aim for MATS and research-engineer roles. That's a great career — it just isn't this one.
This is the most-hired skill inside the labs — information security is nearly half of all lab postings — and almost no learning path treats it as a first-class track. If your background is closer to security than ML, this may be your fastest door in; if it's the reverse, this is the section that makes you rare.
You're done when you've planted a backdoor in a model you fine-tuned — and then built the detection that finds it.
The most durable way to understand an attack is to run it end-to-end from both sides. Planting a data-poisoning backdoor and then building its detection gives you the attacker's and defender's view at once — and an artifact almost no candidate has.
- 🔨
[reference implementation]— a hands-on poisoning/backdoor lab. The literature is deep but hands-on teaching resources are thin; the repo will ship a runnable exercise.
You're done when you can take a deployment you own and place it on RAND's SL1–SL5 scale, naming the specific attack vectors it's open to.
Model weights are the crown jewels, and there's now a canonical framework — cited by the labs themselves — for reasoning about how hard they are to steal. Learning to place a real system on that scale is exactly the threat-modeling a lab's security team does.
- RAND — Securing AI Model Weights — 38 attack vectors, five security levels (SL1–SL5), 167 measures. (Link may block automated fetchers; open it in a browser.)
You're done when you can name what an untrusted model file can do to your machine, and have blocked it in a pipeline you own.
A downloaded checkpoint is untrusted code, and treating it as inert is how supply-chain attacks land. Understanding — and blocking — what a malicious model file or dependency can do is basic hygiene the job assumes you already have.
- OWASP Top 10 for LLM Applications (2025) — supply-chain and data-poisoning entries.
- CyberSecEval (within PurpleLlama) — benchmarks for insecure-code and related risks.
This runs in parallel with everything from §2 onward — you apply while you learn, not after. Your milestones are your portfolio; this section turns them into a job. Don't save it for the end.
You're done when a stranger can see, in under a minute, one eval you built and one break you found.
Hiring managers skim. A pinned repo with a clear README, a link to your registered eval, and a screenshot of your leaderboard entry does more than a page of prose. Make your artifacts findable before you make more of them.
You're done when you've applied to a remote, part-time program — the door that doesn't require quitting your job or moving countries.
The field has entry points that don't demand relocation or a career break. Apply to one now, with whatever you've built so far; these programs are designed for exactly the transition you're making, and applying early costs nothing and calibrates you fast.
- SPAR — part-time, remote research program; the lowest-friction starting point.
- Anthropic Fellows Program — funded, ~4 months, covering adversarial robustness, AI control, and AI security among other areas.
- ARENA (funded, in-person, London) and MATS (stipended research fellowship) — higher-commitment, higher-signal.
You're done when you've turned a public result — a leaderboard break, a registered eval, a bug bounty — into a conversation with someone who hires.
Your §2 and §3 artifacts are not just learning; they're introductions. A Gray Swan placement routes into paid engagements directly; a registered eval or a DEF CON poster is a reason for someone to reply to you. Use the artifact as the opener.
- Gray Swan Arena → its private, paid red-teaming network.
- AI Village @ DEF CON — an open poster track on adversarial attacks against agents; in-progress work is welcome.
You're done when you can re-read the three job postings from M0.1 and tick off requirements with links to what you built.
You started by turning three job postings into a checklist. You finish by returning to it and checking boxes with evidence. When most of the list is ticked with links, you're not "trying to break in" — you're a qualified applicant, and you can prove it.
You don't need to be a policy expert. You need to understand why your evals exist — what obligation turns a lab's safety work from a hobby into a legal and commercial requirement. One milestone, then back to building.
You're done when you can explain what a Frontier Safety Framework commits a lab to — and point to the clause that makes your eval someone's obligation rather than someone's hobby.
Frameworks like Anthropic's Responsible Scaling Policy, and regulation like the EU AI Act, are what make evals and red-teams mandatory rather than optional. Understanding the mechanism — capability thresholds triggering required safeguards — tells you why your work has a budget and a deadline, and lets you talk to the people who set them.
- Anthropic — Responsible Scaling Policy — the first such framework; 11+ companies have since adopted similar ones.
- EU AI Act — GPAI Code of Practice — rules in force since August 2025; enforcement from August 2026.
- AI Safety Atlas, chapter 4 — governance in context.
This roadmap lives or dies by one rule: every node is a checkable milestone, not a keyword. See CONTRIBUTING.md for the quality gate. In short — a milestone must state an artifact (You're done when …), and any resource you add must be open, live, and best-in-class. Suggestions welcome via the issue templates.
Content is licensed CC BY-SA 4.0; the code in scripts/ is MIT (see scripts/LICENSE). Share and adapt freely with attribution.