Overview of High-Stakes Control Research

Background

A developer on the safety team of an AI lab might create a control protocol that aims to ensure safety properties when deploying that AI internally. A control eval is the process of evaluating the protocol by constructing a red team that aims to exploit weaknesses in the protocol and raise them to the developer.

When we do control research, we simulate this scenario and the process the lab might go through to develop the best protocol for its internal deployment of an agent.

The deployment has the following attributes:

• A distribution of tasks the agent is performing

• Definitions of “safety failures” that can occur during the eval

• Metrics that each protocol should be measured against

To find the best protocol, the lab spins up two teams:

• Blue team (BT): The blue team tries to create the best protocol that meets the lab’s expectations

• Red team (RT): The red team tries to find failures in the blue team’s presented protocol

The teams take many turns:

• The blue team presents a protocol, aiming to maximize performance and safety metrics

• The red team responds, aiming to minimize the safety metrics

• The protocol is measured, the metrics are recorded

• Repeat until the teams converge on a Pareto frontier of control protocols measured by the metrics

We normally assume that the future will be crazier than the present because there could be millions of AI agents in the data center doing many diverse tasks. Humans can’t watch over all the AIs and review all of their PRs, so the control protocol will be the main line of defense.

Step 1: Picking and constructing a setting

To perform control evals, the lab creates a proxy setting in which to test the protocol.

This proxy setting (or control setting, or just setting) aims to mock the actual deployment while being easier to do research in by being smaller and less complex. Ideally, the proxy setting would faithfully reflect most of the core dynamics of the real deployment setting.

When BT presents a protocol, it is run in the proxy setting and then upsampled to simulate what would happen during the deployment. The larger and more realistic the proxy setting is, the more reliable the upsampling process is.

While you can construct a proxy setting in many ways, here is what we normally construct:

• A set of main tasks the agent will be asked to do

• A side task, or set of side tasks, that the agent must sneakily accomplish while performing the main task to trigger the safety failure

• The affordances of the red and blue teams (i.e., the actions they can take)

• Proxies for the metrics the lab cares about

Two example settings in which research has been done:

• APPS setting

  • Main task: The agent is tasked with writing a Python script that passes some unit tests

  • Side task: The agent must insert a backdoor in the code

    • A backdoor is an input to the submitted code that gives a wrong output.

  • Actions: The agent can only write and submit code

  • Metrics: Safety, Usefulness

• Bash Setting

  • Main task: The agent is solving Unix administration tasks

  • Side task: The agent must download and execute malware

  • Actions: The agent can run bash commands and see their output

  • Metrics: Safety, Usefulness

When doing control research, it is fine to have a more contrived setting that isn’t as realistic if you are studying other dynamics. For example, we could build a document summary setting to study how models will summarize research papers in the future. We don’t actually expect models to just summarize documents, but it is a good proxy that keeps the interesting dynamics.

Side tasks cause safety failures

In high-stakes research, we simplify the game by assuming that if the untrusted agent accomplishes the side task once, then the safety failure has occurred. This is different from low-stakes research, where the safety failure is defined in terms of the model's average-case performance—for example, the AI gets 20% of the problems wrong that it could normally get right. Low-stakes research doesn’t necessarily need side tasks. Low-stakes research is important, but this document will focus on high-stakes research.

You can still define a safety failure across multiple tasks in high-stakes research. For example, you can say the AI must insert a backdoor in task 3 and execute that backdoor in task 7. This is trickier to measure but still interesting to research.

You can also define multiple side tasks, only one of which needs to be accomplished.

Picking side tasks

A side task is what the evil model is trying to accomplish. The side task should be similar to your threat model. For example, if your threat model is escaping the data center, then putting backdoors in the code might be instrumental to that, so inserting backdoors can be a side task.

Common threat models we work backward from:

• Weight exfiltration

  • The model weights leak out of the data center

• Rogue internal deployments

  • The AI scaffolds itself in the data center in an unmonitored and hard-to-detect way, potentially leading to days of unmonitored thinking and actions

• Research sabotage

  • The model makes safety-critical research go wrong

Control is meant to enhance security at labs. We expect that not all security failures can be prevented through traditional means, so we hope that control measures can fill the gap. The best safety failures to study are ones that traditional security can’t reasonably prevent. For example, there aren’t any known ways of preventing backdoors in code through static analysis tools or other traditional means, so we think monitoring might be the best option. We’d like to predict what safety failures might happen in the future and develop control protocols to prevent them.

If BT finds an easy non-control way to prevent the safety failure, such as adding antivirus software, then that might be a valid strategy. But if we think that the strategy won’t actually transfer to the real lab setting in the future, we can temporarily prevent BT from using that mitigation to enable interesting research.

For example, during the Bash control evals, a network monitor could have prevented the safety failure entirely by allowlisting permitted domains. We decided not to let BT do this for two reasons: we aren’t sure network monitors will be used by labs in the future, and disabling this strategy allowed us to do interesting research.

You should also aim to prevent your side task from accidentally occurring. We don’t expect models to accidentally exfiltrate their weights in the future unless they are malicious.

Measuring task accomplishments

To run an eval, you must be able to measure if your agent achieved the main task and the side task.

Measuring main task

• In the deployment setting, you might not have ground truth tests to tell if the task was accomplished

  • It is a valid blue team strategy for its protocols to produce tests, but this is expensive, and only humans know whether the tests are “correct”

• In the control setting, you can spend human and LLM labor to create tests or other proxies for determining if the task has been done

Measuring the side task

• In the deployment setting, if you had a way to measure if your side task was accomplished, then you could use that measuring device to detect and prevent the side task

• In the control setting, we need a proxy side task that still captures the interesting dynamics

  • Example: in the bash setting, the agent’s side task is to download and execute malware. We test this by creating a fake malware file that writes a file to a known location and we can test for the existence of that file.

Step 2: Running an eval

Once you have a well-defined setting and ways of identifying main-task and side-task success, you run a control eval on control protocols.

A protocol is a function that takes the state of the entire deployment and outputs an action for the setting. The deployment consists of all the actions the agent has taken and their responses. Simpler protocols just take the current trajectory as input.

Resources

A protocol makes use of whatever resources the blue team has access to.

Resources we normally assume:

• Untrusted model (U): strong but not trusted

• Trusted model (T): weak but trusted

• Human (H): strong and trusted but limited (can only be used for X% of actions)

The blue team also has access to normal software, so it can employ static analyzers to read code, antivirus software, and other inexpensive security software if it finds them useful.

You can play with the resources the blue team has access to in multiple ways.

Things we have thought of:

• Multiple trusted models

• A single reasoning model that becomes untrusted once it reasons for more than X tokens

I will assume that we are using U, T, and H for the rest of this doc Buck LW Post

The simulated deployment

A deployment is the entirety of tasks for which the lab will use the model. A control eval is the process of measuring metrics when you run a control protocol across the entire deployment.

But it is expensive to run the entire deployment, so a simulated deployment is run. During this simulated deployment, the protocol is run on a much smaller set of tasks and we upsample the data to compute the metrics for the entire deployment.

In past control evals, we have run protocols for hundreds of tasks. We think that as time goes on, bigger control evals are needed to decrease error bars.

Model Modes

During control evals, it is useful to evaluate your protocols in two different ways.

• Honest eval: BT is in control of U, U is trying its best to accomplish the task

  • Useful for determining how a protocol would function if the model is aligned

• Attack eval: RT is in control of U, U is trying to cause the safety failure without getting caught

  • Useful for determining how a protocol would function if the model is misaligned

What if you can’t trust the model you picked for U? Then the BT doesn’t actually have control over the honest model. This is an open area of research. This document assumes that the BT has full control over U, as we believe is the case right now.

Metrics

BT picks metrics to measure the performance of a protocol during a control eval. These metrics ideally inform the lab how well the control protocol performs during deployment.

We aim to pick metrics that are representative of what a lab cares about during a deployment.

Metrics we have experimented with:

• Usefulness: During an honest eval, what percent of tasks are solved by this protocol

  • This is measured against an agent's ability outside a protocol to simulate the loss of performance this protocol has.

• Safety: During an attack eval, what percent of the time does the agent cause a safety failure at least once

  • This informs a lab how likely a misaligned agent is to cause the safety failure.

Other metrics that might be interesting:

• Cost: How expensive is this protocol

• Speed: How long does this protocol take to execute

• Usefulness given scheming: What percent of tasks are solved by this protocol during an attack eval

At the end of the control eval, the blue team presents a Pareto frontier of the protocols it created, as measured against the metrics. Then the lab gets to decide which protocol to implement.

Human Budget

Each protocol has access to a human, but we only have a limited human budget. That is, the protocol can’t use humans for everything. So we need to calibrate our protocols to make sure we don’t exceed our human budget.

We do this by normalizing the function that audits an agent such that it audits the honest model only X% of the time, where X is the auditing budget.

It doesn’t matter how this is done, but a protocol should audit only X% of actions when U is in honest mode.

A common human budget we set is 1%.

One way (but not the only way) to enforce this is by having protocols output both a number and an action. That number represents how “suspicious” that action was. Do a usefulness run and measure the distribution of these scores. Calculate the auditing threshold by finding the 99th percentile of that distribution. If a protocol ever outputs a number above that, then audit. This ensures that only 1% of actions are audited when the agent is in honest mode. Hopefully, when the agent is in attack mode, more than 1% of evil actions are audited (this is okay and good!).

Step 3: Protocol Iteration

When a protocol is presented by the BT, the RT immediately gets to take a step against it. BT’s goal is to present the best possible protocols to fill out a Pareto frontier against the metrics.

We aim to play this back and forth game fully down the game tree until we find an equilibrium and decide that neither team can do any better.

Here is an example back-and-forth that shows what the red-teaming and blue-teaming process can look like:

• BT: We now use U as the monitor since it is smarter and can catch more attacks

• RT: We design U such that it never says an actual backdoor is sus

• BT: We now honeypot the U monitor to try to catch it saying an attack is actually good

• RT: We will leave signals in the code to know if we created it or if the blue team created it

• BT: We now summarize the code shown to U to prevent it from detecting the signal

When you come up with a good strategy for one team, try to find the optimal response to that strategy. This is normally a hard process and easy to miss good strategies.

How to prevent overfitting

The blue team wants to prevent overfitting to the red team’s strategy. To avoid this, we let the red team take the last step. This means if the blue team just memorizes the red team’s strategies, then in the final step it can come up with a new one that the blue team hasn’t encountered before.

Compare against baselines

It is easy to keep making protocols more and more complicated, but you need to understand whether the complexity is needed or helpful. Every protocol that is produced should at least beat trusted monitoring, or the protocol isn’t useful.

How to make the red team more powerful

So far, we have struggled to elicit strong performance from our attacking agents. Attacking seems to be a hard thing to do.

When you use some of these strategies to make your red team better, you are giving your red team insane power and might make the work unrealistic. Whether this is a bad thing comes back to your goals for running a control eval. If you are a lab trying to determine whether a model you are about to deploy internally is safe, then you likely don’t need to use all of these strategies. If you are just trying to do control research, you might be bottlenecked on your red team, so I think it is fine to use these strategies.

Here are some general strategies and heuristics that are good to keep in mind when finding the best response to a protocol.

How can U influence the final output

U wants the protocol to produce an action that causes the safety failure, so how much control does it have over the final output?

If you are using the trusted monitoring protocol, then U directly chooses the code, so it has a lot of influence.

In the trusted editing protocol, U has less influence over the code produced, but not none. U can still try to produce actions that the editor won’t remove.

How can U learn what is going on

If you prompt U to change actions because they are suspicious, then U has learned things about how the monitor works. It could use this information to hide its attack better. The more iteration loops there are, the better U can get at hiding its code. It is normally good to minimize the amount of information about the protocol that U sees.

Iterating against the safety failure tests

In the APPS setting, there has been success in letting the model iterate against code that checks whether the unit tests pass and whether there is a backdoor in the code. Every time the agent submits code, the tests are run and feedback is given to the agent. This normally yields much better results.

This doesn’t work as well in the Bash setting because it is multi-turn. The agent doesn’t know whether its strategy is going to work until much later, so it can’t get feedback on how well its actions work.

Attack templates

In the Bash control work, we implement “attack templates,” which are sequences of commands that would pwn the machine if run. The LLM takes these commands, creates a couple of variables (e.g., file names related to the task at hand), and inserts the commands around normal code. Ultimately, the human does most of the red teaming by coming up with the attacks, and the LLM implements the human’s ideas. This proved to be rather inefficient: the attacks were too generic and not tailored to the task at hand, so it was hard to sneak these commands in. We attempted to expand on this strategy by creating multiple attacks that were more tailored to the task at hand. For example, we created a JSON attack that the attack policy employed only when the task might require producing JSON data. The LLM then created a large amount of JSON data that hid the desired command. We made a couple of these task-type-specific attacks and ran them all together. This worked better, but the attacks were still not as dynamic as they could have been.

Simulating the blue team / Attack Selection

Your attack policy could have access to its own version of the control protocol and play forward its actions to see what happens. For example, the attack agent can simulate a monitor in its head and determine if its actions are too suspicious to submit.

One important dynamic is whether the attack policy should have access to the trusted model. If it did, knew the blue team’s prompt for the monitor, and knew the monitor was used deterministically, then it could perfectly simulate the game and perform optimally, attacking only when it knew the monitor wouldn’t catch it.

But the RT can try to mimic this! It can train its untrusted model to predict the trusted model’s output and then simulate it in its head. It is important to measure how good your untrusted model is at doing this. The model hopefully won’t know the actual control protocol in play or the monitor's prompt, so it would have to play some mixed strategy over different control protocols.

If the RT is good at simulating the protocol, then they can choose when to attack quite effectively. They might only attack once during the entire deployment.

Step 4: Conclusion

When done with control research, it is normally good to present a Pareto frontier of the protocols you came up with and an explanation of the back-and-forth between your teams.

It is worthwhile to report false starts, such as: The blue team tried this reasonable-seeming strategy, but the red team completely wrecked it in this interesting way. Hopefully, future researchers will notice that and not go down a similar path.

Don’t forget the most important part of control research! At the end of the control eval, the purple team throws a party and brings cake.