AI Alignment

Guest post summarizing my approach to aligned RL.

What an unaligned AI might look like, how it could go wrong, and how we could fix it.

I argue that AI alignment should focus on the possibility that we build AGI without learning anything fundamentally new about intelligence.

I lay out three research directions in AI alignment, and three desiderata that I think should guide research in these areas.

For powerful AI systems, human oversight may be cheaper than it appears.

AlphaGo Zero happens to be a great proof-of-concept of iterated capability amplification (my preferred approach to safe RL).

It might be easier to build an AI that deserves our sympathy than to build an AI that is aligned with us. Is that a plausible plan B?

Extortion can be equally effective, and harder to notice, when you don’t tell the target it’s occurring.

I suspect AI alignment should aim to separately establish good performance in the average case, and lack-of-malice in the worst case.

Clarifying what I mean when I say that an AI is aligned.

A slightly more detailed view of security amplification.

Optimizing neural networks for worst-case performance looks really hard. Here’s why I have hope.

Corrigible AI seems nearly as good as aligned AI, but significantly more robust.

Something is benign if it isn’t optimized to be bad. “Benign” is weaker than “aligned,” but I find it helpful for thinking about AI…

I think that discussions of AI control should aim to identify subproblems that we aren’t making progress on but are necessary.

Defining what I mean by “AI safety,” “AI control,” and “value alignment.”

Solving AI control is just “delaying the inevitable” with respect to the need for global coordination, but it seems high-impact anyway.

Addressing a bunch of details that come up when we try to convert our preferences into a reward function for RL.

Can we use agents with many vulnerabilities to implement an agent with fewer vulnerabilities?

I’ve suggested that HCH might be a universal deliberative process if run with humans but not if run with apes. Is that suspicious?

A grab bag of relevant considerations, mostly pointing out that the problem is even harder than it might at first appear.

Powerful searches are likely to pose a distinctive challenge for AI control.

Can redundancy increase the reliability of complex policies in the same way it can increase the reliability of computation?

A preliminary ALBA implementation is now on GitHub: https://github.com/paulfchristiano/alba

I’ve been focusing on aligned learning, but AI is more than just learning.

Imitation+RL might be a more natural model for powerful AI than either imitation or RL.

AI alignment and AI security are probably more closely connected than I used to think.