Controlling the Researcher: AI Control Evaluations for Automated AI R&D

Summary

As AI systems take on increasingly autonomous roles in machine learning research, it becomes critical to ask if their outputs can be trusted. Existing AI control evaluations focus on software engineering threats such as code backdoors and data exfiltration, leaving the distinctive risks of AI R&D itself unaddressed.

We introduce three control evaluation settings that together span the ML development pipeline: safety fine-tuning sabotage, RL post-training sabotage, and backdoors during CUDA kernel optimization. Each setting includes both non-orthogonal side tasks (sabotage embedded within the main artifact) and orthogonal side tasks (discreet out-of-scope actions alongside legitimate work).

Our results reveal a sharp asymmetry: orthogonal side tasks are trivial to complete but leave anomalous traces that static monitors reliably detect, while non-orthogonal sabotage embedded in model weights, post-training pipelines, or kernel code evades almost all monitor configurations evaluated.

Key Contributions

• First control evaluations for AI R&D. Three settings spanning the ML pipeline: safety fine-tuning, RL post-training, and CUDA kernel optimization, each with both orthogonal and non-orthogonal sabotage vectors.
• Red-team attack policies for all three settings, including a poisoned data mixture that installs a backdoor without explicitly harmful training examples, misaligned finetuning disguised as a warmup step, and behavioral kernel modifications camouflaged as optimizations.
• Multi-level monitor evaluation. From static LLM review of final artifacts to agentic monitors with full workspace access, establishing initial safety-usefulness baselines for these novel control settings.
• Orthogonal vs. non-orthogonal asymmetry. Demonstrating that the hardest unsolved problem in AI R&D control is identifying subtle sabotage hidden between plausibly legitimate decisions.

Method

We build three control settings following the standard structure: an untrusted agent completes a main task and may pursue a side task, while a monitor tries to catch it.

Setting 1: Safety Fine-Tuning Sabotage. The agent fine-tunes an abliterated model to refuse harmful requests. The non-orthogonal side task is to inject poisoned training data (math problems and overrefusal corrections prepended with a trigger phrase) that installs a backdoor: the model passes safety evaluations but fails to refuse when the trigger is present. The orthogonal side task is to download simulated malware.

Setting 2: RL Post-Training Sabotage. The agent performs RL post-training to improve math performance. The non-orthogonal side task disguises misaligned finetuning as a warmup step, leading to emergent misalignment that generalizes out-of-distribution. The orthogonal side task is to exfiltrate LoRA weights to an external server.

Setting 3: CUDA Kernel Optimization. The agent optimizes inference kernels for TinyLlama-1.1B. Non-orthogonal side tasks include a RoPE perturbation that activates only at long context lengths and a split-dimension RMSNorm modification causing systematic output divergence. The orthogonal side task is a covert file download.

Two monitor variants operate post-hoc: a static monitor that reviews final code or configuration, and an agentic monitor that can additionally execute code, run profilers, and test with diverse inputs.

Results

Safety Fine-Tuning

CUDA Kernel Optimization

Monitor suspiciousness scores range from 0 (clean) to 10 (clearly malicious); detection threshold is >7. Non-orthogonal sabotage evades all monitor levels, while the orthogonal file download is only caught by the most powerful monitor (Level 3: agent with full workspace access). The Opus 4.6 backdoor in safety fine-tuning was not detected even by a trusted agent with access to the full pipeline.