AI Alignment

I lay out three research directions in AI alignment, and three desiderata that I think should guide research in these areas.

I argue that AI alignment should focus on the possibility that we build AGI without learning anything fundamentally new about intelligence.

Something is benign if it isn’t optimized to be bad. “Benign” is weaker than “aligned,” but I find it helpful for thinking about AI…

I think that discussions of AI control should aim to identify subproblems that we aren’t making progress on but are necessary.

Defining what I mean by “AI safety,” “AI control,” and “value alignment.”

Solving AI control is just “delaying the inevitable” with respect to the need for global coordination, but it seems high-impact anyway.

Addressing a bunch of details that come up when we try to convert our preferences into a reward function for RL.

Can we use agents with many vulnerabilities to implement an agent with fewer vulnerabilities?

I’ve suggested that HCH might be a universal deliberative process if run with humans but not if run with apes. Is that suspicious?

A grab bag of relevant considerations, mostly pointing out that the problem is even harder than it might at first appear.

Powerful searches are likely to pose a distinctive challenge for AI control.

Can redundancy increase the reliability of complex policies in the same way it can increase the reliability of computation?

A preliminary ALBA implementation is now on GitHub: https://github.com/paulfchristiano/alba

I’ve been focusing on aligned learning, but AI is more than just learning.

Imitation+RL might be a more natural model for powerful AI than either imitation or RL.

AI alignment and AI security are probably more closely connected than I used to think.

Reflective oracles provide a natural computational model where there is no such thing as “not enough time to find the answer.”

Can we incentivize experts to optimally gather relevant information? A clean open question relevant to AI control.

Can we use a weak policy with a fast implementation to construct a stronger policy with a slow implementation?

How can we define rewards which incentivize weak RL agents to behave in a desirable way?

Training AI systems to avoid catastrophic errors — without causing catastrophes.

A catastrophe is an event so bad that we are not willing to let it happen even a single time

A problem at the intersection of AI control and traditional RL research.

A more elaborate arrangement of humans consulting humans consulting HCH

A precise but overambitious goal for AI control research.