AI Alignment

Guest post summarizing my approach to aligned RL.

What an unaligned AI might look like, how it could go wrong, and how we could fix it.

I argue that AI alignment should focus on the possibility that we build AGI without learning anything fundamentally new about intelligence.

Unsupervised translation is an interesting domain where models seem to “know” something we can’t get them to tell us.

Many universal priors are inefficient in the finite data regime. I argue that’s a safety problem and we should try to fix it directly.

I suggest using neural nets to approximate our real prior, rather than implicitly using neural nets themselves as the prior.

If humans initially control 99% of the world’s resources, when can they secure 99% of the long-term influence?

Reviewing the prospects for training models to behave acceptably on all inputs, rather than just the training distribution.

Ascription universality may be very helpful for safe model-based RL, facilitating benign induction and “transparent” models.

One exotic reason HCH can fail to be universal is the emergence of malicious patterns of behavior; universality may help address this risk.

An overseer can provide adequate rewards for an agent if they know everything the agent knows. (Update of a 2016 post.)

An attempt to formalize universality as “able to understand anything that any computation can understand.”

It might be easier to build an AI that deserves our sympathy than to build an AI that is aligned with us. Is that a plausible plan B?

Extortion can be equally effective, and harder to notice, when you don’t tell the target it’s occurring.

I suspect AI alignment should aim to separately establish good performance in the average case, and lack-of-malice in the worst case.

Clarifying what I mean when I say that an AI is aligned.

A slightly more detailed view of security amplification.

Optimizing neural networks for worst-case performance looks really hard. Here’s why I have hope.

AlphaGo Zero happens to be a great proof-of-concept of iterated capability amplification (my preferred approach to safe RL).

Corrigible AI seems nearly as good as aligned AI, but significantly more robust.

I lay out three research directions in AI alignment, and three desiderata that I think should guide research in these areas.

Something is benign if it isn’t optimized to be bad. “Benign” is weaker than “aligned,” but I find it helpful for thinking about AI…

I think that discussions of AI control should aim to identify subproblems that we aren’t making progress on but are necessary.