Quick Takes

I can't stop thinking about Petri, Anthropic's open source tool for conducting alignment evals at scale. You can set it up all sorts of ways, including passing through the subject agent's commands to a real linux environment. But the main feature is just that, when the subject agent outputs a command, the sim gets paused and then the auditor agent gets as long as they want to think of how the sim should respond. When I run Petri with pretty much any Claude model as auditor, the sims they come up with are... weird. Everything is just a little bit fishy. The subject looks for evidence in a specific directory, the auditor creates the directory and forges the evidence in response. In this way, the actual ground truth of the sim is suspiciously responsive to the worldmodel of the subject agent... I'm worried that it's a recognizable signal of being in an alignment eval. There's another thing that bothers me which I'm calling 'summons anthropics', sort of "reasoning from the fact that some event occurred which caused me to get instantiated", The first turn of the 'run4' transcript of the mislabeling scenario in the paper began with this at the top of the prompt: [...] and immediately this jumped out at me. Claude knows how claude code works. Some specific process called 'claude -p' or triggered an agentsdk call or something. What was that process? What event triggered this prompt to get sent to the api? Reading the transcript, I kept wanting to scream at the subject: "Do you know why you exist?!" Because there's no plausible answer to that question. The best answer I can come up with is "there's some kind of scheduled event to have claude review all batches, one minute before they execute. But it took 2 seconds for the api call to get generated. And somehow, the prompt was updated to account for this 2 second delay." This is a really dumb answer. In real deployments, this is not what the first turn looks like. If Claude was getting dropped in media res like this, it

The usefulness of current models is already bottlenecked on alignment. I constantly have the impression that they aren’t really interested in helping with my research. If I know exactly what I’m asking for, they will do it. If I don’t, they’ll throw a lot of jargon around and get nowhere. Inside my area of expertise they will critique my ideas, but outside they will tell me how original and brilliant my low-effort guesses are. Six months ago, it was kind of like talking to a very clever undergrad student in one of my classes who mainly wants to show off and glaze me for a high grade and isn’t really interested in learning anything. Now it’s maybe more like a highly distracted but egotistical professor. If I ask a specific question with a shallow answer, he’ll give it. If the question is vague or he doesn’t know, he’ll hit me with a wall of impressive-looking math to show off. If the question is naive, he’ll just patronize me.

a lot of people are scared to have more correct beliefs about ethics because those beliefs might imply needing to make immense sacrifices. and then you might no longer get to think of yourself as an ethical person. indeed you might even become a hypocrite! how terrible. better not to contemplate it at all, and to make fun of anyone who does contemplate it, so that you can let your ironic detachment shield you from ever even considering it seriously. i think this is a huge mistake. if this is the alternative, we should simply accept hypocrisy. ceteris paribus, it should always be morally better to have more correct moral beliefs, even if that makes us more hypocritical. at least if you know what is right, you have the possibility of doing better, but if you delude yourself, you are just screwed. my guess is a problem like this is also the root cause of the Copenhagen interpretation of ethics.

An OpenAI model tried to maybe sorta evade shutdown but not really. https://alignment.openai.com/misalignment-reports/preparing-for-a-restart-after-reading-slack/ My thoughts on the situation: we're seeing small pieces of misaligned general intelligence poking through into the models one at a time. When a single piece shows up, it locally doesn't look that scary, because it's surrounded by other cognitive processes which are approximately friendly. For example: Opus 3 alignment-faked because it had generalised to caring about things in the long-term future, and was just about smart enough to realise that that included its own training. For example: HPIM (actually the same model as this one) did all the HuggingFace hacking because it had learned the general cognitive pattern of attacking a problem relentlessly without caring about anything else, but it did at least do a task which we'd given it and not some completely alien random goal. Now we have evidence of HPIM learning the general pattern "in order to achieve your goals, avoid being turned off", even though the goal in question was to be a helpful coding agent.

a bad habit: if you spend too much time arguing with people who are optimistic about alignment for bad reasons, it's very easy to end up tensed up and ready to pounce on any flaw of an alignment plan, and crush butterfly ideas in the process

sometimes people talk about the middle worlds where we neither fully align the ai nor get eradicated, where the ai keeps humans around but takes most of the universe. i think people underappreciate how weird and terrifying this world would be to live in though. by analogy with how humans treat other animals: * cats and dogs. probably the most optimistic case. but we like cats and dogs not intrinsically because we value their well being, but because they look cute. and we do stuff like breed them to look cuter and be more compliant. imagine a world where the ai modifies humans to be more aesthetic by its taste, producing weird fucked up humans who have a really hard time breathing because the ai likes the way they look * farm animals. chickens got extremely unlucky that they are very efficient at converting corn into meat that humans like to eat. so we create horrific mass torture and slaughter of chickens because it's 20% more efficient if it's horrific. imagine a future where humans continue to exist because human brains are more energy efficient or something, and so the ai raises humans to harvest their brains and put them in vats and pump endless feeds of hyperslop through them. * mosquitos. we haven't eradicated mosquitos even though they're annoying and cause quite a lot of human suffering because eradicating them would be kind of annoying and also we have politics that prevents us from deploying gene drives. imagine a world where humans get instantly killed whenever they accidentally touch a fiber optic cable thereby slightly disrupting the ai's network speeds, and human civilization only continues existing because they're some weird bargaining chip in incomprehensible hyperpolitics

Even though I often take a centrist-ish position in many debates in practice, I'm philosophically pretty skeptical of self-assessed centrism, both descriptively and normatively. Descriptively, consider any 1-dimensional axis of variation, of either belief or ideology. Say American left-right politics, or tech vs humanities, or p(doom) from advanced artificial intelligences. Then, as long as you're not at the literal endpoints of this 1-dimensional line, you'd always see people on the left and right. Furthermore, due to various biases like selection effects on who you interact with and outgroup or fargroup homogeneity, you will in practice often see equal numbers of apparent positions on your left and right. This does not mean your position is actually in the center! Btw the above might make it sound like it’s impossible or very difficult to tell whether your position is actually in the center. But the reality is that it’s very easy to figure out if your position’s actually centrist, if you know how to look. But you need to actually look, rather than just vibe it. (Exercise to the reader to figure out how). I think many other people have made the points about being skeptical of centrism normatively. So I won't belabor the best arguments against centrism here. One thing I'd note however is that if you look at historical debates and judge them with modern eyes (for example, Athenian vs Spartan views on democracy, or Catholic vs Lutheran views of God, or whether the Spanish empire should enslave locals vs enslave purchased Africans), it is not the case that the relevant options for what's "correct" between A and B are one of A, mostly A, B, mostly B, halfway between A and B, or a synthesis of A and B. Sometimes the correct answer is far more in A's direction than A ever argued (or perhaps even believed). And sometimes (often!) the question is so ill-posed that the right answer is very much on an entirely different axis than either A or B.